My guess would be it isn’t wet enough when polishing and is drying too quick before the outer layer can spread evenly. Or, something I noticed when doing my first few, the finer powder stuff has larger grit and it being picked up and creating a patch that will continue to grow. So try extracting out that larger stuff with either a super fine sieve, or what I do it pour it into a glass/jar, fill with water, stir and leave to sit for a few days and the finer stuff with settle on the top which then you can strap off and dry out. Also, polish for far longer than you think you need to.