I think your experience is correct.

Your body keeps a set calorie usage target that it will adjust to. So over long times it will normalise to using a set amount. New exercise will burn extra energy initially, but you then adjust to doing it.

Diet changes have best effect, exercise is needed to stop you losing muscle mass.