As shown in fig. when a spherical cavity (centered at O) of radius 1 is cut out of a uniform sphere of radius R (centered at C), the center of mass of remaining (shaded) part of sphere is at G, i.e. on the surface of the cavity. R can be determined by the equation :

Let M₁ be the mass of whole sphere, and M₂ be the mass of cavity

GC = GP-CP = (2-R)...(i)

${\mathrm{M}}_1=\frac{4}{3}{\mathrm{\pi R}}^3\mathrm{\rho } \mathrm{and} {\mathrm{M}}_2=\frac{4}{3}\mathrm{\pi }\left(1{)}^3\right(-\mathrm{\rho })$

${\mathrm{X}}_{\text{com }}=\frac{{\mathrm{M}}_1{\mathrm{X}}_1+{\mathrm{M}}_2{\mathrm{X}}_2}{{\mathrm{M}}_1+{\mathrm{M}}_2}$

$\Rightarrow \frac{\left[\frac{4}{3}{\mathrm{\pi R}}^3\mathrm{\rho }\right]0+\left[\frac{4}{3}\mathrm{\pi }\left(1{)}^3\right(-\mathrm{\rho })\right][\mathrm{R}-1]}{\frac{4}{3}{\mathrm{\pi R}}^3\mathrm{\rho }+\frac{4}{3}\mathrm{\pi }\left(1{)}^3\right(-\mathrm{\rho })}=-(2-\mathrm{R})$

$\Rightarrow \frac{(\mathrm{R}-1)}{\left({\mathrm{R}}^3-1\right)}=(2-\mathrm{R})$

$\frac{(\mathrm{R}-1)}{(\mathrm{R}-1)\left({\mathrm{R}}^2+\mathrm{R}+1\right)}=(2-\mathrm{R})$

$\left({\mathrm{R}}^2+\mathrm{R}+1\right)(2-\mathrm{R})=1$

Alternative solution:

Sum of the Moment of masses about COM is zero.

$\begin{array}{l}\therefore M_{remaining}\times GC = M_{cavity}\times OC\text{}\\ \Rightarrow \left(M_{total}- M_{cavity}\right)\times \left(GP-CP\right) = M_{cavity}\times \left(CP-OP\right)\\ Since, Mass\propto {\left(radius\right)}^3\text{}\\ \Rightarrow \left(R^3-1^3\right)\left(2-R\right)=1^3\left[R-1\right] \text{}\\ \Rightarrow \left(R^2+R+1\right)\left(2-R\right)=1\end{array}$