If the resultant of the two forces has a magnitude smaller than the magnitude of the larger force, the two forces may

Let $A>B$

$R=\sqrt{A^2+B^2+2AB\cos \theta }$

If $0\le \theta \le 90°$ , then $R>A$
  
In order that $R<A$, then $\cos \theta$ must be negative. It means $\theta$ must be obtuse but not necessarily $180°$  . 
Further, $\left|2AB\cos \theta \right|>B^2\Rightarrow \left|\cos \theta \right|>\frac{B}{2A}$
If vectors are in same direction, then the resultant is sum of the individual magnitudes.
If vectors point in opposite direction, the resultant is difference between their magnitudes.