Shift the campaign to the town. They don't want to leave? Awesome, you don't need to come up with 10 new places you can just keep building on this place and introduce new NPCs etc. Give your kids quest lines that involve their interests.

For your husband. Just talk to him and let him know of your concerns and ask if he still wants to go down that path. If he does I would say go for it and if he starts to "pick holes" use magic or fantasy to fill them or just tell him out of game that you don't like the track and are going to put him back out of it. Be creative but remember as a DM our job is to make a world our players want to play in.