If$5a – b , 2a + b , a + 2b$ are in A.P. and ${(a – 1)}^2 , (ab + 1), {(b + 1)}^2$are in G.P.$, a \ne 0$, then a is equal to

 $2(2a+b)=(5a-b)+(a+2b)\Rightarrow b=2a.$

Also, $(a-1{)}^2(b+1{)}^2=(ab+1{)}^2$

$\Rightarrow (a-1)(b+1)=\pm (ab+1)$

$\Rightarrow (a-1)(2a+1)=\pm \left(2a^2+1\right)$

$2a^2-a-1=2a^2+1$

or $2a^2-a-1=-2a^2-1$

$\Rightarrow a=-2,a=1/4$