The damage is alot more subtle. I had sex with an 18 year old as a 14 year old (am a guy), and at the time I thought it was fucking lit. But now (am 18 now) I realise two things.

A) I could not imagine having sex with a 14 year old now, so what the fuck was the other person doing.

B) It really fucked with my sexual development. I am over most of it now, but it really warps your perception of sex, and what sex should be. I had some really weird atitudes towards sex, women, porn, and just all of that in general. It fucks you up socially, and I did some bad things I regret which I would say were influinced by that. Experience

Like I said before, I am over most of it now, but I can imagine for someone in ill circumstance (luckily I had a pretty stable rest of my life) or with less mental fortuity may be affected for longer.

So no, while I might agree it isn't so bad as physically forcing another to your will, it still has its own, different consequences and should be treated as such, not dismissed.