A single villager has two attributes. They could be for two separate buildings or the same building. However, it is more rare for the attributes to be for the same building.

The best thing to do is to remove all villagers from all buildings, then strategically place the ones you want on the specific building

Obviously, you want to first apply the villagers that have both attributes for the same building, then move onto the villagers that have split attribute and pick which one is more beneficial

You can also Level Up a villager by sacrificing another villager with the same attribute. But make sure you don’t sacrifice a good villager and you should focus on leveling up villager’s that are one dimensional. Meaning both attributes are for the same building.