Answer: 26

Small Hint:

Put $B=(0,0),$ $A=(0,6),$ and $C=(2,0)$

Big Hint:

The center lies on the perpendicular bisector of $AC$ at distance $\sqrt{40}$ from its midpoint

Solution:

Put $B=(0,0),$ $A=(0,6),$ and $C=(2,0).$ The midpoint of $AC$ is $M=(1,3),$ and $AC=\sqrt{40}.$ If $O$ is the center, then $$\begin{aligned} OM&=\sqrt{OA^2-AM^2}\\ &=\sqrt{50-10}=\sqrt{40}. \end{aligned}$$ A vector perpendicular to $AC=(2,-6)$ is $(6,2),$ which already has length $\sqrt{40}.$ Thus the two possible centers are $$\begin{aligned} M+(6,2)&=(7,5),\\ M-(6,2)&=(-5,1). \end{aligned}$$ The pictured notched circle has its center on the side opposite the notch, so $O=(-5,1).$ Therefore $BO^2=(-5)^2+1^2=26.$