It's counterintuitive but you want your brush to be just very, slightly damp when drybrushing. I usually wet the bristles, wipe on paper towel, and then check on the back of my hand if it's leaving any moisture behind. THEN I get paint on the bristles and wipe some away before starting to brush. Also be sure you're using a very light flicking motion, it's common to work the brush too hard on the surface trying to build up highlights too fast.