The area of the triangle formed by pair of lines ${\left(\mathrm{ax}+\mathrm{by}\right)}^2-3{\left(\mathrm{bx}-\mathrm{ay}\right)}^2=0$  and the line $\mathrm{ax}+\mathrm{by}+\mathrm{c}=0$ is 

detailed solution

Correct option is C

The area of the equilateral triangle formed by the lines ax+by+c=0  and the pair of lines passing through the origin is   c23(a2+b2)given lines ax+by+c=0      ……….(1)By splitting the combined equation ax+by2-3bx-ay2=0ax+by+3bx-ay=0a+3bx+b-3ay=0        …………..(2)ax+by-3bx-ay=0a-3bx+b+3ay=0    ………….(3)Solving (1) and (2) vertex is A=cb-3a3a2+b2,-ca+3b3a2+b2Solving (2) and (3) , vertex is 0=0,0Solving (3) and (1) Vertex is B=-cb+3a3a2+b2,ca+3b3a2+b2 Areaofthetriangle OAB=12x,y2−x2y1=c23a2+b2 squareunits