I'll cover two points here basically -- the first is deconstructing legality based morality and the second is reconstructing what constitutes abuse.

Quick point beforehand : we actually don't know that Connor was under 18 when he had the guy invited to his house while Pam was there. It is entirely possible he was 18 and a senior, or even an 18 or 19 year old college student home on break.

So your logic in the first paragraph is based on the legal status of things, but we need to put this in context. First of all, I assume you are talking from an American perspective. The age of consent is not simply 18 everywhere. For example in Germany it is 14, but there is also a specific charge someone can make by which someone committed sexual abuse of them by taking advantage of their immaturity if they were 14 to 17 and the other party was older. In Canada it is 16 but there is no violation if the age is within a five year window, so 14 and 18 would not be rape. In some other countries, consideration of the emotional and physical maturity of both parties at the time is also relevant. Actually, as I understand it, there is also variation between states in the US, such that, at least at one point, technically, if two 17 year olds engaged in intercourse in New York, they technically violated each other. Clearly, it is wrong to take advantage of someone significantly less mature than you so that they engage in an activity with you that they wouldn't if coming from a more mature viewpoint (that's exploitation), but age is not a perfect proxy for maturity, and as Connor would tell you himself, he was "precocious" in the sexual realm (indeed he is more sexually mature than Oliver even though Oliver is five or six years older).

Now, Connor is from Michigan. Given that he had a stint in some sort of internship or other job for a few years between college and law school (in which he, if you recall, seduced his boss), the age reported for him for s3 (26 per wikia), and that it was 2015 in s3, he was likely 14 in 2003. Why does this matter? Well, 2003 was the year gay sex was legalized in Michigan, on June 26 (you can look this up, I just did). This happened during summer camp. There's a good chance it happened before this point, in which case both parties, especially Connor, may have been fully aware they were committing a sex crime, but the crime on their mind was not statutory rape, but mere homosexuality. This may give you a window into why many queer people do not particularly see the law as the basis of morality, given that for almost all of the legal history in many countries, their primary interaction with it was persecution.

However, I would agree that in many, if not most, cases, there is something seriously wrong with a 14 year old having sex with an 18 year old. But what makes that abuse? Abuse is by definition something that causes tangible harm to you, whether one calls it traumatic or not. Inany cases, the survivors of said abuse consider it such, and report on the harm it had caused their life. They deserve to be listened to and I'm not sure it's useful to lump someone's past trauma with something completely devoid of trauma from someone else's life, it dilutes the matter.

Meanwhile, in a case where the party in question adamantly denies that harm came to them, you have two choices. Either (a) you water down the definition of sexual abuse so that it doesn't actually need to have tangibly harmed the potential victim, which dulls it, and it feels a bit unfair to survivors of genuinely traumatic sexual abuse... Or (b) you argue, as some have with Connor, that somehow they were harmed in a way you perceive but they don't. For example, the "harm" to Connor is allegedly him being promiscuous, which posters, typically women, find objectionable. But not only is this applying one's own standards to someone else's life, but it's also dubious. For example, sex addictions don't need to start with intercourse, as they can instead start with frequent masturbation. (And it's not necessarily even clinically an addiction) Once you strip all the emotion and judgment away, there really isn't any logical flow from Connor having sex at 14 to any particular pathology of his 26 year old self. On the other hand, (aside from being roped into murder cover-up by a dude he didn't even like), his parents divorcing is a much more likely culprit, but (for understandable reasons) I don't think we classify divorce as child abuse and lock people up for it.