Answer: 40

Small Hint:

Multiply the two equations: the cross terms become the constants $12\mathrm{i}$ and $20\mathrm{i},$ leaving an equation in the single unknown $zw$

Big Hint:

The quadratic in $zw$ needs $\sqrt{416 - 210\mathrm{i}};$ find it by setting $(a + b\mathrm{i})^2 = 416 - 210\mathrm{i}$ with integers $a$ and $b$

Solution:

Multiplying the two equations gives $$\begin{aligned} &zw + 12\mathrm{i} + 20\mathrm{i} \\ &\quad {}- \frac{240}{zw} = (5 + \mathrm{i})(-4 + 10\mathrm{i}) \\ &\quad = -30 + 46\mathrm{i}, \end{aligned}$$ so $zw - \frac{240}{zw} = -30 + 14\mathrm{i}.$ Setting $v = zw$ yields $v^2 + (30 - 14\mathrm{i})v - 240 = 0.$

By the quadratic formula, $v = -15 + 7\mathrm{i}$ $\pm \sqrt{(15 - 7\mathrm{i})^2 + 240}$ $= -15 + 7\mathrm{i}$ $\pm \sqrt{416 - 210\mathrm{i}}.$ Writing $(a + b\mathrm{i})^2 = 416 - 210\mathrm{i}$ requires $a^2 - b^2 = 416$ and $ab = -105,$ which gives $a + b\mathrm{i} = \pm(21 - 5\mathrm{i}).$ Hence $v = 6 + 2\mathrm{i}$ or $v = -36 + 12\mathrm{i},$ with $|v|^2 = 40$ or $1440.$

The smaller value is attained: $z = 1 - \mathrm{i},$ $w = 2 + 4\mathrm{i}$ satisfies both equations with $zw = 6 + 2\mathrm{i}.$ So the smallest possible value of $|zw|^2$ is $40.$