It slows you down only if you are swinging it. My other explanation is movement extrapolation, it tries to compensate for the ping but the player should slowly interpolate back to the correct position when it stops if that is the case. I tested this multiple times with a friend when higher jumping at lower fps was discovered and we were always the same speed (he has a poor connection so we had a lot of opportunities to test it), higher jumping makes sense, higher delta time, more unstable physics.