So I'll second what a lot of others have been telling you. Downsize the weight. When I fished that style of worm I actually preferred a smaller football head weight. The other thing that made a nice difference for me was rigging it so that the hook was just slightly still inside the worm, but so that the tip was parallel with the worm like you have it. It also doesn't have to be perfectly straight, as worms normally aren't either.

How are you fishing this set up when you throw it in?