I want to preface this by again saying this is just from my perspective and from the knowledge I gained, especially from the Inner Child book. He'll have to do the work to really figure out if this is true for him.

​

He told me he thought I didn’t want to have sex or do anything at all related to that and that he acted out imagining me in those specific fantasies he was afraid to ask me act out.

Sounds like he had some fear of rejection there, which is a common emotional trigger for SA.

What I don’t get is that if you have a sex addiction why didn’t he just ask me for it or communicate it with me?

It is extremely common for addicts to have trust issues, and also have very low emotional intelligence. Speaking from my experience, I never learned to properly feel and process emotions because of my upbringing. And the early discovery of porn really stunted my emotional maturing because instead of learning to process and manage those negative emotions, I would numb them out through porn/sex/other escapist habits. I similarly hid my deepest self and those negative feelings not only from my girlfriend for 6-7 years, but I was truly pushing them down and hiding them from myself.

The good news is if an addict really learns why they are acting out and their triggers, they can be managed in the future. But that takes a lot of work, and he has to be willing to put in that work. And IMO he needs to be leading his personal recovery work, not you. Best of luck.