Suppose $a^2=5a-8\text{ and }{b}^2=5b-8$, then 

equation whose roots are $\frac{a}{b}\text{ and }\frac{b}{a}$ is

$a, b$ are roots of $x^2-5x+8=0$

$\therefore a+b=5,ab=8$

Now, $\begin{array}{l}\frac{a}{b}+\frac{b}{a}=\frac{a^2+b^2}{ab}=\frac{(a+b{)}^2-2ab}{ab}\\ =\frac{25}{8}-2=\frac{9}{8}\end{array}$

$\therefore$ required equation is 

                                 $x^2-(9/8)x+1=0\text{ or }8x^2-9x+8=0$