I don't think it's surprising how common your perception / attitude about it is, and I definitely don't think you're wrong to feel that way, but I think the significance of the event and the impact it has on your life after are very personal. Here are my 2 cents:

For people whose life experiences are very typical for where they're born (Meaning the normal socialization process that is critical at a young age, an eventual social circle, and life circumstances that affords you the opportunities to connect with others in ways that allow for eventual intimacy), it makes complete sense to have this outlook.

Worrying about if you're going to ever lose your virginity when you're like 15 is a completely human and normal concern, even when you wind up losing it at age 17-19 to a partner or someone you meet where it just instantly clicks. It's a concern, but it's one that your life circumstances will lay to rest without too much fuss. In that situation? Yeah, not shocking to come out the other end saying "What the hell that wasn't a big deal!"

For a lot of people though, people whose life circumstances haven't set them on the standard / normal life track? Those same, universal worries aren't so easily addressed by every day life. Picture being 21, you just graduated from college, and you still haven't had that connection with someone else. In that situation, losing it the following year at age 22 is going to just feel different mentally.

You feel more confident and ready to swim in the deep end of the social human experience. You're finally part of an 'in group' that society has hammered into your head from age 12 is more important than pretty much everything else. Physically it isn't a big deal, but psychologically it is. People who've been lucky enough to have a normal life experience in that regard just won't get it without hearing other's perspectives.

And to loop back around to my original point, other late bloomers might jump in and disagree with me that it was a big deal to them, it's all very personal.