Answer: 578

Solution:

Since $5^2 + 12^2 = 13^2,$ triangles $AEB$ and $CFD$ are right-angled at $E$ and $F,$ and they are congruent (sides $5,$ $12,$ $13$). Extend $EA$ beyond $A$ and $FD$ beyond $D$ until the two lines meet at $G.$

Then $\angle GAD = 90^\circ$ $- \angle EAB = \angle ABE,$ and $\angle GDA = 90^\circ$ $- \angle FDC = \angle DCF$ $= \angle BAE.$ These two angles sum to $90^\circ,$ so $\angle AGD = 90^\circ,$ and triangle $AGD$ is congruent to $BEA$ (equal angles and hypotenuse $AD = BA = 13$). Hence $GA = EB = 5$ and $GD = EA = 12.$

Therefore $GE = GA + AE = 5 + 12 = 17$ and $GF = GD + DF = 12 + 5 = 17,$ with a right angle between them at $G,$ so $EF^2 = 17^2 + 17^2 = 578.$