The resultant of two forces at right angle is 5 N. When the angle between them is ${120}^0,$ the resultant is $\sqrt{13}.$ Then, the forces are

Let, A and B be the two forces

As per question $\sqrt{A^2+B^2}=5$

or $A^2+B^2=\mathrm{25............}\left(i\right)$

and  $A^2+B^2+2AB\cos {120}^0=13$

or $25+2AB\times (-1/2)=13$

or $AB=25-13=12$

or $2AB=\mathrm{24..........}\left(ii\right)$

Solving Eqs (i) and (ii), we get

$A=3N$

and $B=4N$