Solution:

Let the cone have base in the plane $z=0$, center at the origin, and vertex on the $z$-axis. The centers of the three spheres form an equilateral triangle of side $2r$, so one sphere center may be taken at horizontal distance $\frac{2r}{\sqrt3}$ from the cone axis and height $r$ above the base.

In the axial cross-section through that center and the cone axis, the side of the cone is the line $12\rho+5z=60$, where $\rho$ is horizontal distance from the axis. The distance from $\left(\frac{2r}{\sqrt3},r\right)$ to this line must be $r$: $$r=\frac{60-12\cdot\frac{2r}{\sqrt3}-5r}{13}.$$

Thus $(18+8\sqrt3)r=60$, so $$r=\frac{60}{18+8\sqrt3}=\frac{90-40\sqrt3}{11}.$$

Thus, B is the correct answer.