The transformed equation of $3{\mathrm{x}}^2-6\mathrm{xy}+8{\mathrm{y}}^2=8$ when the axes are rotated about the origin through an angle $\mathrm{\pi }/4$ in the positive direction, is

$$\begin{gathered} \mathrm{Given} \mathrm{\theta }=\frac{\mathrm{\pi }}{4}=45° \\ \mathrm{Now} x=X \mathrm{cos\theta }-Y \mathrm{sin\theta } \\ =X \cos 45°-Y\sin 45° \\ =\frac{X}{\sqrt{2}}-\frac{Y}{\sqrt{2}} \\ =\frac{X-Y}{\sqrt{2}} \\ \mathrm{and} y=X \sin \mathrm{\theta }+Y \cos \mathrm{\theta } \\ =X\sin 45°+Y \cos 45° \\ =X \left(\frac{1}{\sqrt{2}}\right)+Y\left(\frac{1}{\sqrt{2}}\right) \\ \mathrm{y}=\frac{X+Y}{\sqrt{2}} \\ \mathrm{Now} \mathrm{transformed} \mathrm{equation} \mathrm{of} 3{\mathrm{x}}^2-6\mathrm{xy}+8{\mathrm{y}}^2=8 \\ \mathrm{is} 3{\left(\frac{X-Y}{\sqrt{2}}\right)}^2-6\left(\frac{X-Y}{\sqrt{2}}\right)\left(\frac{X+Y}{\sqrt{2}}\right)+8{\left(\frac{X+Y}{\sqrt{2}}\right)}^2-8=0 \\ \Rightarrow \frac{3{(X-Y)}^2}{2}-6\frac{(X^2-Y^2)}{2}+8\frac{{(X+Y)}^2}{2}-8=0 \\ \Rightarrow 3X^2+3Y^2-6XY-6X^2+6Y^2+8X^2+8Y^2+16XY-16=0 \\ \Rightarrow 5X^2+10XY+17Y^2-16=0 \end{gathered}$$