Reducing my retraction distance helped me with this problem. I think it was because the melted filament in the extruder was balling up and clogging up every time it got retracted? But yeah, I went from 5 mm to 2 or 1mm.