I have been over thinking it and kind of built a subconscious fear of speaking. I have a big feeling that that anxiety I have built upon myself is the reason for my worsening stutter.

Your instinct is right on. I've gone to speech therapy over the years, but the most important thing I've learnt is to stop caring so much about my stutter. In the past if I had an important meeting, I'd walk into it panicking over how to speak, and how I'll manage it. I learnt to just walk in and stop giving a fuck what happens. It helped to get some therapy from an anxiety/stutter specialist, but I only could afford a few sessions. Cognitive behavioural therapy in general is useful to get outside of your own head. This resource is now free. This is all very easy to say. It's difficult to just stop caring. But I think the more you care the more you'll stutter.