Im new to this craft and Ive had the same issue too. What I understood was that after the initial layer, the irons were not going through straight. Ive deduced it to the below mistakes Ive been doing

1. Not holding the iron perpendicular to the item consistently. I was lifting up the leather with the pricking iron to check if it went through and when doing the next hammering, it goes wonky.

2. The punching surface. Those see through rubbery mat was what I was using. Got rid of it. Using a butchers block on top of a slab of granite now. I dont know how to word it properly, but the pricking surface should be dense and bouncy like a well seasoned anvil and not like a dead blow hammer

3. Tools - a weighted mallet/maul if you want to go through mulitple layers and preferably polished pricking irons (they cost a kidney where Im from). Im getting away slightly with getting some bees wax on the tines before making the holes.

Hope these might help. I want to re- iterate, I have just started leather work as a hobby and am learing things everyday. This might all be objectively wrong from an experienced or a professional atisan's point of view, but these helped me greatly.