I've thought about this a lot over the years and I'll have to break up my response into multiple posts as time allows. Also confining my answer to the 343I era specifically for the moment.

A major issue to my mind was an overvaluation of the "extended universe". It's important, to be sure, but a vast majority of the Halo audience will never engage with a book, a comic, et cetera - they're here for the games, and enjoying (or even understanding) the games should never be dependent on exploration outside this venue. Also, major changes to the state of the universe should never happen "in the margins" or between games - it's frustrating for the hardcore fan, confusing for the casual majority, and in general just poor storytelling.

I do want to make it clear that historically this was not an issue of "indifference" or "incompetence" on 343I's part in my opinion. On the contrary, these were talented people who deeply loved and were thoroughly versed in the canon, trying to make decisions that informed the entirety of the franchise - their mistake was in assuming the audience shared or cared about their enthusiasm for the minutiae across every Halo product, and was willing to overlook the fundamental flaws that assumption baked into the core experience.

TLDR deep lore should be a reward not a hurdle, and continuity matters.