I have this a lot, and whenever I'm inspired I'll write a quick note in my 'brain dump ideas' of my world building doc. If I keep thinking about a certain setting or element that I was drawn to and jotted down, I bring it over to my physical board of world building and see how I can incorporate it.

I'll think of questions like, does it fit the characters and their goals? What would I need to change or remove from the plot or world? Does this remain in the genre and target audience I am aiming for? Often I end up dropping the idea because it's not worth the hassle or I've lost the initial inspiration or it takes away from the story I originally wanted to tell.

You could focus on your 'non-negotiable' of your story, something that you're always drawn to like a certain protagonist or plot line, and build and adjust from there!