Answer: 315

Small Hint:

The vector $(b,37)$ is a rotation of $(a,11)$ through $60^\circ$ or $-60^\circ$

Big Hint:

Use the second coordinate of the rotation first, then compute the first coordinate

Solution:

For a $60^\circ$ rotation, $$37=\frac{\sqrt3}{2}a+\frac{11}{2},$$ so $a=21\sqrt3.$ The first coordinate is then $$b=\frac a2-\frac{11\sqrt3}{2}=5\sqrt3.$$ The opposite orientation changes both signs and leaves the product unchanged. Hence $ab=(21\sqrt3)(5\sqrt3)=315.$