Solution:

Note that the average of $5$ consecutive numbers starting with $x$ is $$\dfrac{5x + 1 + 2 + 3 + 4}{5} = x + 2.$$

This means that the average of $5$ consecutive integers starting with $a$ is $a + 2,$ which we know is $b.$

Furthermore, the average of $5$ consecutive numbers starting with $b$ is $$b + 2 = a + 4.$$

Thus, B is the correct answer.