If a and b are two unit vectors such that a+2b and 5a−4b are perpendicular to each other then the angle between a and b is
450
600
cos−11/3
cos−12/7
Since the vectors a+2b and 5a−4b are perpendicular so
a+2b.5a−4b=0⇒ 5a2+6a·b−8b2=0⇒ 6a·b=3a=b=1⇒a·b=12⇒cosθ=12a=b=1⇒ θ=600