Answer: 578

Small Hint:

Since $5^2 + 12^2 = 13^2,$ angles $AEB$ and $CFD$ are right angles

Big Hint:

Extend $EA$ beyond $A$ and $FD$ beyond $D$ to meet at $G,$ and look for right triangles congruent to $AEB$

Solution:

Since $5^2 + 12^2 = 13^2,$ triangles $AEB$ and $CFD$ are right-angled at $E$ and $F,$ and they are congruent (sides $5,$ $12,$ $13$). Extend $EA$ beyond $A$ and $FD$ beyond $D$ until the two lines meet at $G.$

Then $$\begin{aligned} \angle GAD &= 90^\circ - \angle EAB \\ &= \angle ABE, \\ \angle GDA &= 90^\circ - \angle FDC \\ &= \angle DCF = \angle BAE. \end{aligned}$$ These two angles sum to $90^\circ,$ so $\angle AGD = 90^\circ,$ and triangle $AGD$ is congruent to $BEA$ (equal angles and hypotenuse $AD = BA = 13$). Hence $GA = EB = 5$ and $GD = EA = 12.$

Therefore $GE = GA + AE = 5 + 12 = 17$ and $GF = GD + DF = 12 + 5 = 17,$ with a right angle between them at $G,$ so $EF^2 = 17^2 + 17^2 = 578.$