First count colorings of a grid whose positions are distinguished. Fix the center square as red and list the four edge-middle colors cyclically. Up to a rotation or reflection, the only possible edge patterns are $YRBR$ and $YRBY.$ The first has $4$ placements and $3$ possible cyclic corner strings, $BRYB, BYRB,$ and $BYYB;$ the second has $8$ placements and $2$ possible corner strings, $BYBB$ and $BYRB.$ Thus there are $4 \cdot 3 + 8 \cdot 2 = 28$ colorings with a red center. The center has $3$ possible colors, so there are $84$ labeled colorings.

Now apply Burnside's lemma. The identity fixes all $84$ colorings. No nonidentity rotation fixes a valid coloring. Each of the two reflections across a horizontal or vertical axis fixes $6$ colorings, while each diagonal reflection fixes none. Therefore, the number of colorings up to rotations and reflections is $\dfrac{84 + 6 + 6}{8} = 12.$ Thus, C is the correct answer.