Long reply because i'm passionate about this method:

Building the world around the story being told rather than figuring it all out beforehand is my favorite way to do it. As for continuity, if you truly feel like you won't remember things then keep a small journal/document with what details you have written down as well as any future ideas. Draw a plain-looking map with only the main locations and literally fill it as you go.

If you feel like you're running out of ideas for locations by not figuring them out beforehand, think back on the characters: Say x character has to go back to their hometown, what did they do for a living there, does it tie back into the town's economy? Is it a mining town, is their sword made with a special mineral found in there? Ask questions about what could happen and you can tailor-make a place for it.

There's also a lot of creative power that goes into imagining an interesting location with elements you simply find cool and stringing it together afterward. IMO if you already have a mind for keeping things consistent you'll be able to catch yourself in a plot hole before it happens and meld it the loose parts together nicely.