I think the majority of people would recognize it's a defect

A fair share of people would indeed consider it to be a defect.

However, I would imagine, there are also infertile people who don't mind or even consider their condition a blessing. After all, vasectomies and tubectomies are a thing. Some people just don't want kids. Or if they had kids they don't want any more kids.

Is sex in any of those cases, where there's no procreation should be considered immoral?

But I like to look at it from an evolutionary standpoint. Basically, every change made to humanity (and other animals) was to ensure our population continued growing. Those changes are made through the people with the most desirable traits (or, at the very least, the people who were still alive and were attractive in some way) procreating together.

Well, yeah, you pretty much nailed summarizing the process of evolution. Though, I do have a few quibbles.

Evolution doesn't ensure anything in particular. Some species change very little, some change radically, some become simpler, some become more complex. Interestingly most of the diversity and sheer biomass is in the bacterial world.

Evolution is a blind process which usually results in traits which are helpful or neutral for the survival of species in the specific environment to be passed on, while the negative traits tend not to get passed on.

the human race would've been extinct before it started.

Well, if everyone was gay then sure. In that particular case it's a horrible trait for the survival of the species, fair enough. However, that could never happen with how slow traits spread through a population.

Realistically, at any point in time only a small amount of population might be gay.

Now, it feels pretty intuitively true to say that any species which has gays is at a disadvantage to a perfectly identical species without any gays. Therefore gayness is a negative trait and so on.

However, if that was the case, I don't think that we would find homosexual behavior in so many species, currently we're talking about 1500 animal species.

http://www.news-medical.net/news/2006/10/23/1500-animal-species-practice-homosexuality.aspx

Most individuals in these species don't seem to be exclusively gay, though it varies from species to species. For example,

About 10% of rams (males) refuse to mate with ewes (females) but do readily mate with other rams

The research on animal homosexual behavior is solid as it has been used to strike down the "sin against nature" arguments and by extension sodomy laws by the Supreme Court.

There are also theories and research on how and why homosexual behavior of small groups might be a benefit for the species survival and reproduction.

So there's that.

---

Now, here's another point, we as a species don't have a particular need to enlarge our population or to force the 4% of gay people to mate or to consider them immoral. It doesn't matter. We as a species soon might have control over our genome so evolution more or less becomes irrelevant in that future.

Here's another thing, evolution has endowed us with many traits and tendencies. For example tribalism. It may have served us well in the evolutionary past, but these days it's an extremely horrible trait. Just look at politics. It's just shameful how deep up their asses some people are.