if the coefficients of  $x^7$  in ${\left(a{x}^2+\frac{1}{bx}\right)}^{11}$  and $x^{-7}$  in ${\left(ax-\frac{1}{b{x}^2}\right)}^{11}$  are equal, then $ab=$  

Given expansion is  ${\left(a{x}^2+\frac{1}{bx}\right)}^{11}$  

We know that $r^{th}$  term in the expansion of ${(x+a)}^n$ is

$\begin{array}{c}{T}_{r+1}{=}^n{\text{C}}_r{x}^{n-r}{a}^r\\ \end{array}$

$T_{r+1}={}^{11}{\text{C}}_r{\left(a{x}^2\right)}^{11-r}{\left(\frac{1}{nx}\right)}^r\mathrm{................}\left(1\right)$  

$T_{r+1}={}^{11}{\text{C}}_r{\left(a\right)}^{11-r}{\left(x^2\right)}^{11-r}{\left(\frac{1}{n}\right)}^r{\left(x^{-1}\right)}^r$

$T_{r+1}={}^{11}{\text{C}}_r{\left(a\right)}^{11-r}{\left(x\right)}^{22-3r}{\left(\frac{1}{n}\right)}^r$

And this will contain $x^7$  compare with

${\left(x\right)}^{22-3r}=x^7$

Only if $22-3r=7$  i.e. if $r=5$ .These   value sub in Eqn (1)

Hence $T_6$ $={}^{11}{\text{C}}_5{\left(a{x}^2\right)}^6{\left(\frac{1}{bx}\right)}^5$

                  $={}^{11}{\text{C}}_5\frac{a^6}{b^5}{x}^7$  contains $x^7$  

And given another expansion is  $\left(ax-\frac{1}{b{x}^2}\right),$  

We know that $r^{th}$  term in the expansion of ${(x+a)}^n$ is

$\begin{array}{c}{T}_{r+1}{=}^n{\text{C}}_r{x}^{n-r}{a}^r\\ \end{array}$

The general term is

$T_{r+1}={}^{11}{\text{C}}_r{\left(ax\right)}^{11-r}{\left(-\frac{1}{b{x}^2}\right)}^r\mathrm{.................}\left(2\right)$

$T_{r+1}={}^{11}{\text{C}}_r{\left(a\right)}^{11-r}{\left(x\right)}^{11-r}{\left(-\frac{1}{b}\right)}^r{\left(x^{-2}\right)}^r$

$T_{r+1}={}^{11}{\text{C}}_r{\left(a\right)}^{11-r}{\left(x\right)}^{11-3r}{\left(-\frac{1}{b}\right)}^r$

And this contains $x^{-7}$ compare with  ${\left(x\right)}^{11-3r}$

$\Rightarrow {\left(x\right)}^{11-3r}=x^{-7}$

Only if

$11-3r=-7$  i.e. if $r=6$ .These value in Eqn (2) then we get

Hence $T{\text{'}}_7={}^{11}{\text{C}}_6(a{x}^5{\left(-\frac{1}{b{x}^2}\right)}^6$

$={}^{11}{\text{C}}_6\frac{a^5}{b^6}{x}^{-7}$  contains $x^{-7}$  .

According to problem  given that

${}^{11}{\text{C}}_5\frac{a^6}{b^5}={}^{11}{\text{C}}_6\frac{a^5}{b^6}$  

$\Rightarrow a^6{b}^6=a^5{b}^5\Rightarrow ab=1$