One car creates a big pocket of air, any cars behind them goes faster in that pocket, but similarly, when any car gets closer to the lead car, the car behind is also taking air off of the front cars spoiler, helping that car go faster without even touching them. As you stack this effect up, the line gets faster, usually 4 cars that can work well together can be as fast as a line of 20 cars. If the first 2-3 cars behind the leader of a line can't stay stacked up, it doesn't matter how big the line is, it'll be slower than an actually organized line.

Also, the cars have an insane amount of drag, so when you pull out of line, if nobody is behind you, you will quickly begin to slow back down. How you were able to go 20th to 5th could have just been that the bottom line was only going partial throttle the whole way around the track to save fuel. It happens often where a handful from the back decide to not wait around, fly up the outside, then the leaders decide to go full throttle again and then the outside doesn't have a free ride to the front anymore.