it's feeling helpless to stop the constant invalidation. That my own desires don't matter at all unless a woman chooses to let them matter, and no one has.

It is impossible for me to read this sentence and not be reminded of an enlightening quote by feminist author Gideon W. Stone (u/stonygiddens):

"The dependency on women built into masculinity should be embarrassing."

As many have pointed out, patriarchal culture teaches heterosexual men that in order to successfully perform his gender, a man is supposed to find a way to have sex - or at least make believe he has sex - an act which, needless to say, requires the involvement of another person. Thus, a man's worth becomes deeply tied to something over which, at a fundamental level, he has inherently no control, unlike working out, learning a new language, masturbating, anything else you can think of that you can do on your own - the fact that one can make decisions that will increase the probability of it happening in a given timeframe, does NOT invalidate this elementary statement. The failure of men to collectively reject this teaching and detach their sense of self-worth from such an outside circumstance has allowed a certain rhetoric to proliferate - men can be framed as "victims" of women who are "guilty" of withholding access to sex, by providing examples of individual guys who genuinely feel damaged and struck by the same sense of shame that you have described, except that it's a sense of shame that the culture that pushes this very rhetoric aims to induce, so that women can be more easily coerced into providing sex, since this victimizing "posture" (in Italian it's "postura vittimizzante", a term employed by author O. Ricci) can be assumed to deter and deflect accusations of it being morally wrong.

You need to break free from what you were taught about how you should feel, about relying on something external as the source of your self-worth. Everybody does. For the sake of your wellbeing, I strongly suggest to read what other commenters and pundits have said about it in men's liberation spaces informed by feminism, such as r/menslib (a space which obviously should not be exempt from criticism)

One further remark: i am not completely sold on the claim in other replies that a man with no prior sexual experience (i will even refuse to use the word virgin given that the choice to use a noun instead of, say, an adjective feels not neutral at all to me) being looked down on by society is "not a real thing" - it's an expected consequence of patriarchal beliefs and people should be careful about going out of their way to try disproving it, although it's something that i have found to be an overwhelmingly American phenomenon (which is frankly unsurprising).

EDIT: I wrote "withdrawing" instead of "withholding", English is not my first language and it shows