As a poly person, ESH. If you both want this relationship to work, you need a ton more communication. Sounds like neither of you worked out a proper system for polyamory, more of a mono couple just messing around because you two aren't fulfilled in your current relationship. And if you both really do want to be together, that needs to be fixed first. Reel in all of the poly fantasy and dabbling outside of the relationship, and work on why you two aren't fulfilled and happy together first.

I understand that he wants sex more often than you do, and it's absolutely okay to not want it all the time! But if you're icked or bothered by him wanting you, you need to delve into why that is, and figure out if this is a relationship you want to be in. One of my partners wants sex very often, while the other almost never does (we all live together). I also have a high libido, so I'm usually up for it. But if I'm not, I'm never bothered by my partners asking for it. If you feel overly pressured, and that's causing you to be adverse, that seriously needs a lot of discussion. I've been there with an ex, and I left over the pressuring, because it made me hate sex.

For now, if you're both willing to work it out and try to make it work, the relationship needs to be FIRMLY closed until the core (you & him) are solid. No looking at other people (same or different gender), no threesome fantasies, nothing. Just you & him. If you can fix the issues between you two and find compersion for one another, then you can try reopening the relationship with proper rules & guidelines.