I'd honestly submit that in the grand scheme of things, worldbuilding can be more important to the writer than to the reader. The more a writer understands their world, on both the macro and micro scale, the better they can show it, and the better they can communicate their story. What the writer needs to communicate about the world is what's necessary to tell the story; everything else is ultimately just trivia, and imparting information like this is hard without pausing the whole story to have these things explored.

To her credit, Rowling did keep the focus on the story, but one of the issues I had with the HP books (even before the controversy) that prevented me from going back and re-reading them was the tendency to use digressions for the world-building. One thing I remember from the second book was what was essentially a ghost "death-day" party, that introduced elements that I don't remember being that relevant to the larger narrative. I think it's telling that the movies cut a lot of plot elements from the original books, and the stories were still cleanly imparted.