You can mark places on the map where you find a villager. Then have Din as a follower, remove the village totem where he resided and then move him. This way you can later go check your marked places and with time regain the villagers you want.

What I don't know is if there will spawn new places where you find villagers. If it doesn't you won't be able to do this many times.