Solution:

Applying $f$ to $f(f(x)) = f(f(f(x)))$ repeatedly shows the condition means that $f(f(x))$ is a fixed point of $f$ for every $x.$ So the elements organize into levels: a nonempty set of $i$ fixed points, then $j$ elements whose image is a fixed point (but which are not fixed), and the remaining $5 - i - j$ elements, each of which must map to one of the $j$ middle elements.

For given $i$ and $j$ there are $\binom{5}{i}$ choices of fixed points, $\binom{5-i}{j}$ choices of the middle level, $i^j$ maps from the middle level to the fixed points, and $j^{\,5-i-j}$ maps for the rest. Summing $\binom{5}{i}\binom{5-i}{j}\, i^j \, j^{\,5-i-j}$ over the valid pairs (all $i \ge 1,$ $j \ge 1,$ plus the identity case $i = 5$) gives $$20 + 120 + 60 + 5 + 60 + 240 + 80 + 60 + 90 + 20 + 1 = 756.$$