Solution:

Multiplying the three equations gives $$(xyz)^2 = (-80 - 320i)(60)(-96 + 24i) = 80 \cdot 60 \cdot 24 \,(-1 - 4i)(-4 + i) = 240^2 (16 + 30i).$$ Since $16 + 30i = (5 + 3i)^2,$ we get $xyz = \pm 240(5 + 3i).$

Dividing $xyz$ by each given product yields $$x = \frac{xyz}{yz} = \pm(20 + 12i), \qquad y = \frac{xyz}{zx} = \pm(-10 - 10i), \qquad z = \frac{xyz}{xy} = \pm(-3 + 3i),$$ with matching signs. Hence $x + y + z = \pm(7 + 5i),$ so $(a, b) = \pm(7, 5)$ and $a^2 + b^2 = 49 + 25 = 74.$