Solution:

Subtracting the second equation from the first gives $xy - yz + 4z - 4x = 0,$ which factors as $(y - 4)(x - z) = 0.$ So $y = 4$ or $x = z.$

If $y = 4:$ the first equation becomes $4x + 4z = 60,$ so $x + z = 15,$ and the second becomes $4z + 4x = 60$ again while the third gives $zx = 44.$ Then $x$ and $z$ are roots of $t^2 - 15t + 44 = (t - 4)(t - 11),$ so $x \in \{4, 11\}.$

If $x = z:$ the first equation reads $x(y + 4) = 60,$ so $y = \frac{60}{x} - 4,$ and the third reads $x^2 + 4y = 60.$ Substituting, $$x^2 + \frac{240}{x} - 16 = 60 \quad\Longrightarrow\quad x^3 - 76x + 240 = 0 = (x - 4)(x - 6)(x + 10),$$ so $x \in \{4, 6, -10\},$ each with real $y$ and $z.$ Hence $S = \{-10, 4, 6, 11\}$ and the sum of squares is $100 + 16 + 36 + 121 = 273.$