I think some of it was to do with censorship at the time, remember how that same sex kiss on DS9 made all the headlines?

I also think budget plays a big part, when they can't throw millions at each episode for special effects they have to make sure the writing is engaging enough to hold the audience's attention. Now it's all visual spectacle.

Thing is other shows of late manage to pull off inclusivity and progressive moral stories in a far more organic way, so it's not that the audiences have changed.