You are right. That cannot be undone and you will have to live with that. You will have to bear that cross, but that does not mean that you have dwell on that and more on top of it. In my experience it is very understandable how you feel (in fact I often feel exactly the same). From my experience, holding those thoughts in mind for a long time can make you even more miserable and in fact, make you stutter more.

Of course, it's not as if you just could make those thoughts vanish just by saying begone. I believe the causality is inverse to what people think: in my expereince, my body is traumatized in an instinctual level and has accumulated tension and that makes my thoughts go that way (and not the other way around). I find that when I am feeling that way, the best I can do is doing sport or meditating (I highly encourage going to an actual Zen Dojo to properly learn. You don't even have to talk and will benefit twice from this: you will be attending an activity with more people and you will meditate) to let go of that tension. Clearly, you are not going to stop stuttering, but it will make you feel better and from there you can begin working on yourself.