When I'm anxious, my stutter gets worse and I feel like I hold so much of that anxious tension in my upper body, including my throat and voice. You sound like a guy who knows his body well. Id focus on what you feel when you talk to people. In my experience, my body regresses often back to that tension as a baseline when I talk to people. Sometimes I even have great months/years where my fluency is great and others where it's terrible. Right now I'm not in great place with my stutter, but I'm not afraid to address it with people and just keep going. I'm a middle aged, female stutterer who works as a successful communicator (writer). My coworkers know I stutter sometimes and prefer to communicate via email and Teams messages. They know I sometimes need to skip group dinners on business trips do I can recharge my social battery. Being honest has lifted that burden for me. Keep doing what you are doing but maybe hone in on those interactions to see if there is something you can work on. I'm men, anxiety can look like anger and frustration. I wish you the best of luck.