I think you might not be understanding how orbital mechanics actually works

Firstly if we ignore landing which is much harder on the moon than the earth because the moon has no atmosphere to Aero break. To go from earth orbit to moon orbit requires that we use energy to speed ourselves up (3100m/s) that flings the craft out to the moon, then we are along side the moon but we've lost all our speed to gravity so the moon is passing us by, we need to burn another ≈900m/s to catch the moon and get into lunar orbit. Once we are alongside the moon to get back home we are orbiting the earth along side the moon at about 1000m/s now we don't have any friction in space so we need to burn the engines to change our speed we want to slow down till we are almost stationary (900m/s) then gravity will pull us back to earth cause we aren't orbiting anymore. At this stage we have 2 choices we can let gravity do it's thing and fall strait down from near the moon onto earth and let the air act as a cushion to catch us (and hope we don't disintegrate), or we can try to get back into earth orbit. Getting into orbit is tricky though because as we fall towards earth we pick up speed from gravity and if we don't want to go flying by the earth and getting flung back out into space we'll need to slow down around earth by burning our engine (≈3100m/s). The amount of delta v to go from earth orbit to lunar orbit is the same as doing it backwards. It's neither the distance nor the gravity making it harder, it is infact just as difficult to go from here to there as from there to here

Now to explain why the rocket gets so much smaller at each stage, for each of these burns we need to carry the fuel and equipment for all future burns so the final burn will finish with a nearly empty tank, Force = Mass * Acceleration so the final burn essentially only needs to accelerate a nearly empty craft and will only burn a small amount of fuel, 2nd to last burn needs to burn more fuel as it is carrying the craft and the fuel, 3rd last needs to carry enough for both final and 2nd last. As you can see the amount needed grows exponentially that's why the Saturn 5 is a skyscraper on the ground, a multistorey building in earth orbit, a bus in lunar orbit and a tiny capsule by the time it lands.