Answer: 480

Solution:

The diagonals of a rhombus are perpendicular bisectors of each other, so $C = -A,$ $D = -B,$ and $OA \perp OB.$ Let line $BD$ have slope $m,$ so $B = (x, mx)$ with $$x^2\left(\frac{1}{20} - \frac{m^2}{24}\right) = 1,$$ i.e. $$x^2 = \frac{120}{6 - 5m^2},$$ which requires $m^2 \lt \frac{6}{5}.$ Then $BD^2 = 4(x^2 + m^2x^2)$ $= \frac{480(1 + m^2)}{6 - 5m^2}.$ Line $AC$ has slope $-\frac{1}{m},$ so it meets the hyperbola only when $\frac{1}{m^2} \lt \frac{6}{5},$ that is $m^2 \gt \frac{5}{6}.$

On the interval $\frac{5}{6} \lt m^2 \lt \frac{6}{5},$ the quantity $\frac{480(1 + m^2)}{6 - 5m^2}$ is strictly increasing in $m^2:$ as $m^2 \to \frac{5}{6}$ it tends to $480 \cdot \frac{11/6}{11/6} = 480,$ and as $m^2 \to \frac{6}{5}$ it grows without bound. Hence $BD^2$ takes exactly the values in $(480, \infty)$ and never equals $480.$

The largest number that is less than $BD^2$ for every such rhombus is therefore $480.$