If the system of equations ,x + y + z = 0,ax + by + z = 0,bx + y + z = 0,has a non-trivial solution, then

From first and third equation (b – 1)x = 0If  b≠1,  x = 0In this case  y + z = 0, by + z = 0⇒  (b-1)y=0  ⇒y=0Thus , z = 0 That is, if b ≠1, then the system has only trivial solution.∴b=1From first and second equation  (a – 1) x = 0.If a≠1, x = 0 and y + z = 0In this case the system has infinite number of solution If a=0 all the three equations become identical to x + y + z=0  and the system has infinite number of solutions. b=1⇒(b-1)2=0⇒b2=2b-1