If $x=a\cos \theta b\sin \theta$ and $y=a\sin \theta -b\cos \theta$, then the value of $y^2\frac{d^{2}y}{d{x}^2}-x\frac{dy}{dx}y$ is

The given functions are  $x=a\cos \theta b\sin \theta$ and $y=a\sin \theta -b\cos \theta$

Calculate the first derivative of equation $x=a\cos \theta b\sin \theta$ with respect to $\theta$,

$\Rightarrow \frac{dx}{d\theta }=\frac{d}{d\theta }\left(a\cos \theta b\sin \theta \right)$

$\Rightarrow a\frac{d}{d\theta }\left(\cos \theta \right)b\frac{d}{d\theta }\left(\sin \theta \right)$

$\Rightarrow -a\sin \theta b\cos \theta$

$\therefore \frac{dx}{d\theta }=-a\sin \theta b\cos \theta$

Then,

Calculate the second derivative of equation $y=a\sin \theta -b\cos \theta$ with respect to $\theta$,

$\Rightarrow \frac{dy}{d\theta }=\frac{d}{d\theta }(a\sin \theta -b\cos \theta )$

$\Rightarrow a\frac{d}{d\theta }\left(\sin \theta \right)-b\frac{d}{d\theta }\left(\cos \theta \right)$

$\Rightarrow a\cos \theta b\sin \theta$

$\therefore \frac{dy}{d\theta }=a\cos \theta b\sin \theta$

And then calculate $\frac{dy}{dx}$,

$\Rightarrow \frac{dy}{dx}=\left(\frac{dy}{d\theta }\right)\left(\frac{d\theta }{dx}\right)$

$\Rightarrow \frac{a\cos \theta b\sin \theta }{b\cos \theta -a\sin \theta }$

Convert $\cos \theta$ into $x$ and $y$ terms,

Multiply $x=a\cos \theta b\sin \theta$ with $a$

$\Rightarrow ax=a^2\cos \theta ab\sin \theta ....\left(1\right)$

Multiply $y=a\sin \theta -b\cos \theta$ with $b$

$\Rightarrow by=ab\sin \theta -b^2\cos \theta ....\left(2\right)$

Subtract equation $\left(1\right)$ and $\left(2\right)$,

$\Rightarrow ax-by=a^2\cos \theta b^2\cos \theta$

$\therefore \cos \theta =\frac{ax-by}{a^2{b}^2}$

Convert $\sin \theta$ into $\mathrm{x}$ and $\mathrm{y}$ terms,

Multiply $x=a\cos \theta b\sin \theta$ with $b$

$\Rightarrow bx=ab\cos \theta b^2\sin \theta ....\left(3\right)$

Multiply $y=a\sin \theta -b\cos \theta$ with a

$\Rightarrow ay=a^2\sin \theta -ab\cos \theta ....\left(4\right)$

Add equations $\left(3\right)$ and $\left(4\right)$,

$\Rightarrow bx+ay=\left(a^2{b}^2\right)\sin \theta$

$\therefore \sin \theta =\frac{bx+ay}{a^2{b}^2}$

Converting $\frac{dy}{dx}$ in terms of $x$ and $y$,

$\Rightarrow \frac{a(ax-by)b(bx+ay)}{b(ax-by)-a(bx+ay)}$

$\Rightarrow \frac{x\left(a^2{b}^2\right)}{y\left(a^2-b^2\right)}$

$\therefore \frac{dy}{dx}=\frac{x\left(a^2{b}^2\right)}{y\left(a^2-b^2\right)}$

Calculating second derivative of $\frac{dy}{dx}=\frac{x\left(a^2{b}^2\right)}{y\left(a^2-b^2\right)}$,

$\Rightarrow \frac{d^{2}y}{d{x}^2}=\frac{d}{dx}\left(\frac{dy}{dx}\right)$

$\Rightarrow \frac{d}{dx}\left(\frac{x\left(a^2{b}^2\right)}{y\left(a^2-b^2\right)}\right)$

$\Rightarrow \frac{a^2{b}^2}{a^2-b^2}\left(\frac{d}{dx}\left(\frac{x}{y}\right)\right)$

$\Rightarrow \frac{a^2{b}^2}{a^2-b^2}\left(\frac{1}{y}-\frac{x\left(\frac{x\left(a^2{b}^2\right)}{y\left(a^2-b^2\right)}\right)}{y^2}\right)$

$\Rightarrow \frac{a^2{b}^2}{a^2-b^2}\left(\frac{y^2\left(a^2-b^2\right)-x^2\left(a^2{b}^2\right)}{y^3\left(a^2-b^2\right)}\right)$

$\therefore \frac{d^{2}y}{d{x}^2}=\left(\frac{a^2{b}^2}{a^2-b^2}\right)\left(\frac{y^2\left(a^2-b^2\right)-x^2\left(a^2{b}^2\right)}{y^3\left(a^2-b^2\right)}\right)$

Substitute $\frac{d^{2}y}{d{x}^2}$ and $\frac{dy}{dx}$ in the expression $y^2\frac{d^{2}y}{d{x}^2}-x\frac{dy}{dx}y$

$\Rightarrow y^2\left[\left(\frac{a^2{b}^2}{a^2-b^2}\right)\left(\frac{y^2\left(a^2-b^2\right)-x^2\left(a^2{b}^2\right)}{y^3\left(a^2-b^2\right)}\right)\right]-x\left(\frac{x\left(a^2{b}^2\right)}{y\left(a^2-b^2\right)}\right)y$

$\Rightarrow \left(\frac{a^2{b}^2}{a^2-b^2}\right)\left(\frac{y^2\left(a^2-b^2\right)-x^2\left(a^2{b}^2\right)}{y\left(a^2-b^2\right)}\right)-\left(\frac{x^2\left(a^2{b}^2\right)-y^2\left(a^2-b^2\right)}{y\left(a^2-b^2\right)}\right)$

$\Rightarrow 0$

For equations $x=a\cos \theta b\sin \theta$ and $y=b\cos \theta -a\sin \theta$,the value of $y^2\frac{d^{2}y}{d{x}^2}-x\frac{dy}{dx}y=0$

Therefore, the correct answer is option $\left(3\right)$.