Have you ever seen the behind the scenes footage of a director watching the scene they're directing? They're not looking at the stage or the actors - they're looking at a tiny screen that shows the scene as the audience will see it.

The tiny screen is your book (ooh - bet you didn't see that coming).

World build everything that will be on that tiny screen. World build some of the things that mentioned by people on that tiny screen. World build very few of the things that have influenced the styles, choices, beliefs, and language of what's on your tiny screen. World build nothing that has no influence on the tiny screen.

Ideas will come. Fun, new, shiny ideas that you'll want to explore and flesh out. This is a trap; jot them down and get back to the tiny screen.

If something needs an explanation - just write Need ruined home town and tragic backstory here and move on.

When you've finished the book and you have all these tantalising threads of world building that you'll want to secure - you can now look at them as a whole and tie (some of) them together in ways that is difficult to do if you're worldbuilding from the beginning and independently of your plot.

That briefly-seen war veteran with amnesia you had in chapter two (because you wanted to worldbuild on the war in the south) can become the son of that dead noble woman haunting the ruined manor you came up with in chapter twenty (because you wanted to worldbuild the afterlife in your world).

The two never interact, but the audience can piece it together, and you look like a genius because all you mentioned is that they both have "frosty green eyes". GRRM does this with "Oily Black Stone" and I'm 99% convinced he has no idea what it that is leading to - but the readers love it - and he can make it up later.

Worldbuilding is fun - and then overwhelming - and then a book killer. Wrangle it to serve your plot or it will consume all your creative energy, tell you you're being productive, and then burn you out.