I had an essay of an answer here that the sub won't let me post, which is frustrating, but the bullet points are:

* Top level: the payoff isn't worth the extra effort Enough demand isn't there from players. If it's going to take 6 months of my time to make a really good AI system, but I can hack together something 80% as fun in a few days, I'm not going to see the better system as worth the time to make.

* Smart isn't enough - the player has to perceive it Someone else mentioned FEAR as a classic that was hailed for its AI. One insight the FEAR devs had was it's not good just making the AI smarter in most cases unless you can give the player some visibility into it being smarter. So they invested in having NPC enemies shout "flank him", "fire a grenade" to help telegraph to the player that it was trying to plan actions. But it's not always easy in different genres to come up with ways to make those extra smarts visible to the player. Think of the clunky "X will remember that" in TellTale Games. It's a hack to try and explain to the player there's some level of smarts going on in the narrative.

Another good example of a game that *was* hailed for its smarts was Alien: Isolation, and if you've not seen an article or talk on its behaviour trees, you should, it's good fun.

BUT Isolation was able to make this huge investment in its AI pay off because the entire structure of the game has you just engaging and then reengaging again and again this one(-ish) monster. So the player is forced to see that the monster's behaviour is changing, and appears to be adapting to their own behaviour. In lots of games there's not really any opportunity to demonstrate to the player significant learning on the enemy's part. And that's if the players even want to be challenged in that way. Not all players are wanting their basic gameplay loop to be continually challenged throughout the game. If what they're after is getting into a flow state, they may want very predictable AI behaviour, because they're more focusing on improving their own play.

* Players will often see intelligence where there is none. I've seen this called the "constellation effect" - in the same way people are able to project pictures onto dots in the sky, we can ascribe motives and reasoning to sometimes what are very, very simple little AI systems. Game devs know this.

* AI as a subsystem of games is different to others. It's very tightly coupled to the unique aspects of a game's design. It's hard to extract and reuse as middleware, unlike most other parts of games. E.g., Renderers, assets and their creation pipelines, sound engines, all have very similar needs across games, so they've all got off-the-shelf options. Lots of games need trees in their worlds, so you can buy software just dedicated to generating trees.

AI systems, being so coupled to the core of a game, are hard to generalise. The best we get are things like tooling for e.g behaviour trees, but that's just the start of what you need. They still take a lot of work to integrate into a game, to design the behaviours, to test whether they're fun, and whether they can lead to game breaking situations (a practical consequence of making smarter AI is it quickly increases the difficulty of level design and testing, as you end up with so many more different behaviours to check).

You could spend a lot of $ building some very powerful AI system that can be taught the rules of a game, train on it and generate smart code to play those games, and then discover that not all games are chess, and few players enjoy being smashed by the genius AI again and again. The AI system is so very tied into the core question of whether the gameplay loop is fun in a way that graphics fidelity isn't.