The gravitational force between m₁ and m₂ has magnitude
[tex]F_{1,2} = \dfrac{Gm_1m_2}{x^2}[/tex]
while the gravitational force between m₁ and m₃ has magnitude
[tex]F_{1,3} = \dfrac{Gm_1m_3}{(15-x)^2}[/tex]
where x is measured in m.
The mass m₁ is attracted to m₂ in one direction, and attracted to m₃ in the opposite direction such that m₁ in equilibrium. So by Newton's second law, we have
[tex]F_{1,2} - F_{1,3} = 0[/tex]
Solve for x :
[tex]\dfrac{Gm_1m_2}{x^2} = \dfrac{Gm_1m_3}{(15-x)^2} \\\\ \dfrac{m_2}{x^2} = \dfrac{m_3}{(15-x)^2} \\\\ \dfrac{(15-x)^2}{x^2} = \dfrac{m_3}{m_2} = \dfrac{60\,\rm kg}{40\,\rm kg} = \dfrac32 \\\\ \left(\dfrac{15-x}x\right)^2 = \dfrac32 \\\\ \left(\dfrac{15}x-1\right)^2 = \dfrac32 \\\\ \dfrac{15}x - 1 = \pm \sqrt{\dfrac32} \\\\ \dfrac{15}x = 1 \pm \sqrt{\dfrac32} \\\\ x = \dfrac{15}{1\pm\sqrt{\dfrac32}}[/tex]
The solution with the negative square root is negative, so we throw it out. The other is the one we want,
[tex]x \approx 6.74\,\mathrm m[/tex]
