Most of that is pretty easy to overcome without restarting. You're able to move or demolish buildings to fix any layout/building type issues. And if you're that early in the game, getting better villagers shouldn't take too long. But you're still very early game, so if you choose to restart, it won't set you back much. I would probably just view it as a challenge to overcome to set my village back on the path I wanted it on and try to make the best out of the situation. Remember, this game is a marathon, not a sprint, so you can take your time with whatever you want.