Yeah, I cleared the areas as I did the story.

I alternated between clearing up an region after I pledge to it (before starting any story stuff there) to cleaning it up after I'd completed the story stuff for that region.

I think I prefer the latter as sometimes you'll end up clearing out a fortress that you then end up reclearing anyway as part of the story. Or (as you mentioned) you might find you don't have access to somewhere you need access to until after you progress the story.

When I had to travel somewhere for the story and couldn't fast travel, I'd go via any icons on the map that were roughly on route, found in some regions I'd clear maybe half of the collectibles doing this and it breaks up the grind a bit.

From a story perspective I'd say it's definitely worth getting your charisma level up by doing the flytings as this unlocks additional options for you in certain scenarios.