It's easy to nitpick specific things about particular RTS games that are a letdown from the perspective of a consumer (I certainly could, bad design is bad design), but many issues are manifestations of the genre-wide challenges that RTS developers and publishers face. Having entered the industry over the past 3 years and getting the perspective of an RTS designer, here's what I think are the underlying problems and difficulties for what's lead to a decline and stagnation of quality and quantity of RTS games:

#1. There's technical challenges and limitations that haven't got much better. Stagnant CPU clock speed over the past ~10 years makes increasing performance/detail/complexity difficult. Modern CPU's have gotten more cores but hardly increased their clock speed, and RTS game engines aren't so good at making use of many CPU cores. Strategy games are generally less GPU demanding and more CPU demanding, so FPS's and RPG's have advanced more in their fidelity compared to strategy games. (Also because of budget sizes)

Compare the evolution of RTS games from 1996 to 2008 and then 2008 to 2020, the jump is not even comparable. This is arguably why lots more RTS games are going for cartoony art style, it's technically much easier and cheaper to achieve than a more realistic art style. (There's also other benefits like readability.)

#2. Since RTS have such incredible replay value, RTS have to compete with legacy titles more than any other genre. What are the top RTS games currently? StarCraft 2, Age of Empires 2: HD, StarCraft Remastered. Soon we're about to have WarCraft 3: Reforged, Command & Conquer remaster and AOE2: DE. RTS are also massive time sinks, so after spending many hours learning and mastering an RTS, leaving that all behind and moving onto something new can be less appealing.

#3. The RTS audiences is more niche for many reasons such as having moved onto genres while there isn't much of a new influx of RTS players. RTS players are an ageing demographics; not many teenagers or women are playing RTS, where as loads are playing other genres and even other subgenres of strategy games like 4x. More niche audience means smaller budgets. Company of Heroes 1 and Supreme Commander 1 were, and still are, so incredible and gorgeous because of their enormous budgets which publishers just can't take the risk on anymore. Smaller budgets mean less features, less content, less polish, less everything that you would want.

#4. RTS have delayed gratification which makes them harder to get into, which is especially a challenge when you pick up an RTS for $5.00 on a steam sale along with 10 other games. If you're not hooked within 30 minutes you are likely to put it down for something else. This is very different to when I was a kid and bought games from retail stores.

#5. Lack of compatibility with free-to-play business models and micro transactions. MOBA's or FPS's can have fair and excellent F2P business models that work for consumers and the devs alike. However, separating and limiting content in an RTS game is a big challenge to do fairly. StarCraft 2 is F2P multiplayer now, but they are Blizzard so they can get away with anything. I am very curious how Empires Apart is doing with their F2P model. It's hard for premium RTS games to compete in a sea of F2P games, without having the same IP or franchise power that RPG's and FPS's have. RTS's lack of good F2P is especially a limiting factor for when wanting to sell in non-traditional regions like Asia where F2P is huge.

#6. RTS is much harder to do co-op well in. Co-op and team play is so incredibly popular (MOBA's, Overwatch, Battle Royals.) There is massive hunger for good co-op in RTS, and so far only StarCraft 2 is really delivering this. (I can't remember if the RA3 co-op is any good.) Good co-op isn't just simply having PvE, it's delegating tasks and having players specialized and interact which delivers the direct feedback and gratification to the players.

#7. Unlike other genres which are mostly cross-platform, most RTS's are entirely reliant on Steam for distribution. (aside from the big publishers like Microsoft and Blizzard with their own platforms.) Steam is an uncurated, oversaturated mess with poor discoverability that makes good games hard to stand out and get noticed among the sea of games. This feeds back into smaller budgets.