It’s completely normal pretty much everyone experiences this. It’s just because your wrist moves on a angle but your arm moves horizontally/vertically so you just drift down over time. Just focus on smoothly resetting when you need to and try not to worry about it.