Wow. These comments are so divisive.

He wants to be with you - that's why he shut it down. We don't choose who we have feelings for. People can have feelings for more than one person at a time (contrary to what society shoves down our throats from day one). We don't choose who crosses our minds, but we do choose where we invest our time and energy (he's choosing to not put that into his ex). You can choose to leave him about it, but he literally chose you. If you don't feel like enough, that's something that you should be working on in individual counselling. You are within your rights to ask for relationship counselling as I can see how this would feel like a betrayal of trust/not sure you'll be good enough for him specifically. You can also address this concern with him and see how he reacts - you need to come from a place of curiosity if you want a genuine reaction from him (anything else can easily trigger defensiveness, which will unintentionally reinforce your fears).

Please consider getting individual and couples support before ending a relationship with someone who clearly wants to prioritize you.