$\text{ Given equation }{\mathrm{ax}}^2+2\mathrm{hxy}+{\mathrm{by}}^2=0 ...\left(1\right)$
Let the axes be rotated through an angle $\mathrm{\theta }$ and (X, Y) are the new co-ordinates of (x, y)
$$\begin{gathered} \therefore \mathrm{x}=\mathrm{Xcos}\mathrm{\theta }-\mathrm{Ysin}\mathrm{\theta } \text{ and }\mathrm{y}=\mathrm{Xsin}\mathrm{\theta }+\mathrm{Ycos}\mathrm{\theta } \\ \therefore \text{ Transformed equation of (1) is } \\ \begin{array}{l}\mathrm{a}(\mathrm{Xcos}\mathrm{\theta }-\mathrm{Ysin}\mathrm{\theta }{)}^2+2\mathrm{h}(\mathrm{Xcos}\mathrm{\theta }-\mathrm{Ysin}\mathrm{\theta }\left)\right(\mathrm{Xsin}\mathrm{\theta }+\mathrm{Ycos}\mathrm{\theta })+\mathrm{b}(\mathrm{Xsin}\mathrm{\theta }+\mathrm{Ycos}\mathrm{\theta }{)}^2=0\\ \\ \Rightarrow \mathrm{a}\left({\mathrm{X}}^2{\cos }^2\mathrm{\theta }+{\mathrm{Y}}^2{\sin }^2\mathrm{\theta }-2\mathrm{XYsin}\mathrm{\theta cos}\mathrm{\theta }\right)+\\ 2\mathrm{h}\left({\mathrm{X}}^2\cos \mathrm{\theta sin}\mathrm{\theta }+{\mathrm{XYcos}}^2\mathrm{\theta }-{\mathrm{XYsin}}^2\mathrm{\theta }-{\mathrm{Y}}^2\sin \mathrm{\theta cos}\mathrm{\theta }\right)+\\ \mathrm{b}\left({\mathrm{X}}^2{\sin }^2\mathrm{\theta }+{\mathrm{Y}}^2{\cos }^2\mathrm{\theta }+2\mathrm{XYsin}\mathrm{\theta cos}\mathrm{\theta }\right)=0\end{array} \end{gathered}$$
To remove the ‘XY’ term, XY co-efficient should be zero
$\begin{array}{l}\Rightarrow \mathrm{a}(-2\sin \mathrm{\theta cos}\mathrm{\theta })+2\mathrm{h}\left({\cos }^2\mathrm{\theta }-{\sin }^2\mathrm{\theta }\right)+\\ \mathrm{b}(2\sin \mathrm{\theta cos}\mathrm{\theta })=0\\ \Rightarrow -\mathrm{asin}2\mathrm{\theta }+2\mathrm{hcos}2\mathrm{\theta }+\mathrm{bsin}2\mathrm{\theta }=0\\ \Rightarrow 2\mathrm{hcos}2\mathrm{\theta }=(\mathrm{a}-\mathrm{b})\sin 2\mathrm{\theta }\to \left(2\right)\end{array}$
Case(i)
$\text{ If }\mathrm{a}\ne \mathrm{b}\text{ then }$
$\begin{array}{l}\text{ (2) }\Rightarrow \frac{\sin 2\mathrm{\theta }}{\cos 2\mathrm{\theta }}=\frac{2\mathrm{h}}{\mathrm{a}-\mathrm{b}}\Rightarrow \mathrm{Tan}2\mathrm{\theta }=\frac{2\mathrm{h}}{\mathrm{a}-\mathrm{b}}\\ \Rightarrow 2\mathrm{\theta }={\mathrm{Tan}}^{-1}\left(\frac{2\mathrm{h}}{\mathrm{a}-\mathrm{b}}\right)\Rightarrow \mathrm{\theta }=\frac{1}{2}{\mathrm{Tan}}^{-1}\left(\frac{2\mathrm{h}}{\mathrm{a}-\mathrm{b}}\right)\end{array}$
Case(ii)
If a = b then
$\begin{array}{l}\text{ (2) }\Rightarrow 2\mathrm{hcos}2\mathrm{\theta }=0\Rightarrow \cos 2\mathrm{\theta }=0\\ \Rightarrow 2\mathrm{\theta }=\frac{\mathrm{\pi }}{2}\Rightarrow \mathrm{\theta }=\frac{\mathrm{\pi }}{4}\end{array}$