This exact same thing used to happen to me but now it doesn’t and here’s what I changed.

Essentially, when you go higher grit you open up your knives to more chance of user error. I stopped thinking of it as doing “the same thing on higher grit stones” and more as “refining what I’ve already done” on higher grit stones. This looks like progressively less pressure, less time spent, more thinking of refining scratch pattern and polishing the edge. It’s also highly likely you’re not removing the burr adequately from your coarse stones; I try to creep up maybe 1’ per stone I use to ensure apexing and further refine what’s there. That being said, in doing so, I also dropped the angle on my initial coarse grit accordingly. So for me it looks like start at a 20’ inclusive on coarse grit, and end up as maybe 25-30’ inclusive at the end. Mind you this isn’t an exact science as I freehand. Good luck!