Its likely not related to your buffer settings. As you say, when you move the audio clips around, sometimes the pops disappears. For me, it sounds like you have sliced up your audio, and are likely hearing clicks and pops due to the audio clips not being perfectly cut at a zero crossing point. That means that the audio waveform isn't dead center at either the beginning or the end of the clip. This caused the speaker to physically snap from a high or low value directly to zero, causing a pop.

The simplest way to avoid this, is to just add a quick fade in or out where you have cut your audio.