Consider six point masses with mass m placed at the vertices of a regular hexagon of side l. Now, find the force acting on any of the masses.

From figure,

$AE =\mathrm{AD}=\mathrm{AM}+\mathrm{MC}=2\mathrm{AM}=2\mathrm{lcos}{30}^{\circ }$

$=2\mathrm{l}\frac{\sqrt{3}}{2}=\sqrt{3}l\text{ Similary, }\mathrm{AE}=\sqrt{3}l$

$\mathrm{AD}=\mathrm{AO}+\mathrm{ON}+\mathrm{ND}=l\sin {30}^{\circ }+\mathrm{l}+\mathrm{lsin} {30}^0=2\mathrm{l}$

AB = AF = l

Force on mass m at A due to mass m at B is

$$\begin{gathered} {\mathrm{F}}_{\mathrm{AB}}=\frac{\mathrm{Gmm}}{(\mathrm{AB}{)}^2}=\frac{\mathrm{Gmm}}{{\mathrm{l}}^2}\text{ along }\mathrm{AB} \\ {F}_{AF}=Force on the mass m at a due to the mass m at F \\ =\frac{G{m}^2}{l^2} along AF \end{gathered}$$

Force on mass m at A due to mass m at C is

${\mathrm{F}}_{\mathrm{AC}}=\frac{\mathrm{Gmm}}{(\mathrm{AB}{)}^2}=\frac{\mathrm{Gmm}}{(\sqrt{3}\mathrm{l}{)}^2}=\frac{{\mathrm{Gm}}^2}{3l^2}\text{ along AC}$

$$\begin{gathered} F_{AD}=Force on the mass m at A due to the mass m at D =\frac{Gm.m}{{\left(2l\right)}^2} \\ \Rightarrow F_{AD}=\frac{G{m}^2}{4l^2}, along AD \end{gathered}$$

Net force on mass m along AD is given by

$$\begin{gathered} F_R=2F_{AB}\cos {60}^0+2F_{AC}\cos {30}^0+F_{AD} \\ =2\times \frac{G{m}^2}{l^2}\times \frac{1}{2}+2\times \frac{G{m}^2}{l^2}\times \frac{\sqrt{3}}{2}+\frac{G{m}^2}{4l^2} \end{gathered}$$

       =$\frac{G{m}^2}{l^2}(\frac{5}{4}+\sqrt{3})$