When shell strikes the ball and gets stuck with it, combined body of mass 2m starts to move to the right. Let velocity of combined body (just after collision) be ${\text{v}}_{\text{1}}$. According to law of conservation of momentum.
$\left(\text{m+m}\right){\text{v}}_{\text{1}}{\text{=mv}}_{\text{0}}{\text{\hspace{0.17em} or \hspace{0.17em}v}}_{\text{1}}\text{=}\frac{{\text{v}}_{\text{0}}}{\text{2}}{\text{=3ms}}^{\text{-1}}$ 
As soon as the combined body starts to move rightwards, thread becomes inclined to the vertical. Horizontal component of its tension retards the combined body while trolley accelerates rightwards due to the same component of tension.
Inclination of thread with the vertical continues to increase till velocities of both (combined body and trolley) become identical or combined body comes to rest relative to trolley. Let velocity at that instant of maximum inclination of thread be v. According to law of conservation of momentum.
$\left(\text{2m+M}\right){\text{V=2m.v}}_{\text{1}}{\text{\hspace{0.17em}or\hspace{0.17em}v=1ms}}^{\text{-1}}$ 
During collision of ball and shell, a part of energy is lost. But after that, there is no loss of energy. Hence, after collision, kinetic energy lost is used up in increasing gravitational potential energy of the combined body.
If maximum inclination of thread with the vertical be $\text{θ}$ then according to law of conservation of energy.
$$\begin{gathered} \frac{\text{1}}{\text{2}}\left(\text{2m}\right){\text{v}}_{\text{1}}^{\text{2}}\text{-}\frac{\text{1}}{\text{2}}\left(\text{2m+M}\right){\text{v}}^{\text{2}} \\ \text{=2mg}\left(\mathcal{l}\text{-}\mathcal{l}\text{cosθ}\right) \\ \therefore {\text{cosθ=0.8\hspace{0.17em}or\hspace{0.17em}θ=37}}^{\text{°}} \end{gathered}$$