A non-isolated physical system typically loses information to its environment, and when such loss is irreversible the evolution is said to be Markovian. Non-Markovian effects are studied by monitoring how information quantifiers, such as the distance between physical states, evolve in time. Here we show that the Fisher information metric emerges as a natural object to study in this context; we fully characterize the relation between its contractivity properties and Markovianity, both from the mathematical and operational point of view. We prove, for classical dynamics, that Markovianity is equivalent to the monotonous contraction of the Fisher metric at all points of the set of states. At the same time, operational witnesses of non-Markovianity based on the dilation of the Fisher distance cannot, in general, detect all non-Markovian evolutions, unless specific physical postprocessing is applied to the dynamics. Finally, we show for the first time that non-Markovian dilations of Fisher distance between states at any time correspond to backflow of information about the initial state of the dynamics at time 0, via Bayesian retrodiction. All the presented results can be lifted to the case of quantum dynamics by considering the standard CP-divisibility framework.