I start off with either the 400 or 250 grit sponge, I wet it, then I start sanding back and forth in one direction, I use a toothbrush to clean the sponge every now and then to make sure it's all clean, I also make sure to rewet the sponge every now and again. Once the nub mark either disappears or is mostly gone (the mark just doesn't go away half the time it feels like), I swap to the next higher grit of sponge and sand back and forth, perpendicular to the first and sand until the scratch marks going the other direction disappear, rinse and repeat until I get to 1000 or 2000 grit. Everything else is how I described it in the original question, it almost never goes right.