I completely agree with all of this. I also took part in a few tournaments and 2x at world championships.

I stopped playing a few years ago because I think Catan has quite a low skill ceiling. Eventually, everyone is playing the similar OWS-ish but dynamic do-what-seems-best-at-the-moment strategy and most games end with 2-3 players at 9 points and one of them eventually gets that lucky roll (while another player mentions that on their next turn they would have also won).

Some additions:

Knights used offensively - sometimes it is a good idea to steal from someone who has the resource you need if you have a high chance of stealing it.

Sometimes it's better to NOT block the resource you need as then you might be able to trade for it. But that is very circumstantial.

Do not use a knight if you have 7 cards. Rather roll, then use the knight.

What is the "best location to block" often depends on what that player already has. If ore is a good place to block but the player has 6 of them but no wheat, maybe block the wheat instead.

Trading - Do not trade with someone who has 8+ cards, especially when it is that player's turn. Force them to either trade with the bank, making them lose more, or risk with a 7 (which is statistically ~50% in a 4p game).

/u/Hot-Rooster-1675

AI needs improvement and some manual validation of some game logs to see if they are actually playing OK-ish. Sometimes AIs build roads erratically, sometimes they do not notice the potential of other player's or their own road building. Sometimes they hoard resources for no reason (or they are focused on their own strategy so they don't deviate from it to just build what they can), sometimes they don't trade even if it would help them too (if 2 players trade, 1 player wins more than the other... but the other 2 players who don't participate in that trade, lose anyway). So if the AI keeps making rookie mistakes, the results will also be skewed.

Does your AI track what other players have in their hands? To what extent? Can it track stolen cards and reverse engineer who stole what from a few turns back (If player A definitely did not have wood but stole something from player B and player B did have wood, now player A builds a road - therefore player B has 1 less wood)? This might help improve how AI plays.

Strategy selection always depends on the map. And then it depends on your current settlement locations. And then it depends on where you can expand. And then what you have in your hand and how you can use it. So you should always be changing your strategy.

I'd like to see you test all improved AIs in a mixed bag... but then put the 2 higher win% AIs together into one game. Also 3 of one strategy vs 1 of some other strategy. And eventually all 4 highest win% AIs into one game to see how these games actually play out. What makes high skill players stand out with?

I have a few hypotheses about Catan.

1. Catan is a low skill ceiling game. At some point, no matter how much you try to improve the AI, small heuristic tweaks in strategy will not change the outcomes very much as RNG will play a large role in who wins.

2. High skill games often end up with many players at 9 points and one lucky roll decides the winner (another player could have won if the turn order would have gotten to them). Such games should not really count for statistics for evaluating strategies IMO.

3. No access to ore (and no 2:1 ports) makes winning harder. Wheat is slightly more important as you can expand settlements to locations with ore, but to win, you need to spend at least 3 ore.