Answer: 736

Solution:

A triangle with sides $4$ and $10$ is determined by the included angle $\theta,$ and its area is $\frac{1}{2} \cdot 4 \cdot 10 \sin\theta = 20\sin\theta.$ When $\theta \gt 90^\circ$ the triangle is obtuse, and this case produces exactly one triangle for each area $s \in (0, 20).$

When $\theta \lt 90^\circ,$ the third side satisfies $c^2 = 116 - 80\cos\theta,$ and the triangle is obtuse only if the angle opposite the side of length $10$ is obtuse (if $c$ were the longest side, its opposite angle $\theta$ would be acute, making the triangle acute). That requires $4^2 + c^2 \lt 10^2,$ i.e. $116 - 80\cos\theta \lt 84,$ i.e. $\cos\theta \gt \frac{2}{5}.$ Then $\sin\theta \lt \frac{\sqrt{21}}{5},$ so this second family exists exactly for $s \lt 4\sqrt{21}.$

For $s \lt 4\sqrt{21}$ there are two non-congruent obtuse triangles (their third sides differ), while for $4\sqrt{21} \le s \lt 20$ only the obtuse-$\theta$ triangle exists: at $s = 4\sqrt{21}$ the acute-$\theta$ candidate degenerates to a right triangle. For $s \ge 20$ there are none. Hence $[a, b) = [4\sqrt{21}, 20)$ and $a^2 + b^2 = 336 + 400 = 736.$