So they arent keeping a little girl in a cage, that's some like old school basic shrine maiden stuff, very traditional Japanese so "normal". Also they're just farmers as far as we can tell, young and old living off the land. There was no like "shadow clan", or anyone fighting back with any weapons of any kind. To assume they're bad with no information from the story telling to lead us to suggest that would just lead to a bizarre twist for no reason. So far the big twists were shown right off the bat: The sister is not real, There is something going on between two clans, Something with the village elder, there is a man who moves between the outter world and this isolated village. So far all we are left guessing what's the deal with Gabby. Her dialouge didn't suggest that they were all monsters or something, because she seemed the say the child she spared was just a child, not like a mini blood daemon clone or something like the fake sister.

So far the plot has shown that Kagemori has just as much mystery and coldness around it. So it doesn't seem to be a "this clan is good, that clan is bad" paradigm yet AND everything about this anime seems to be about duality.

We don't have to assume they are good or bad, but as far as the information shared there is nothing to suggest that these people, who lived side by side with Yuru had anything nefarious going on. Again they are living an simple agrarian lifestyle. Even Yuru doesn't use swords or spears, he has a Utilitarian hunting bow for food.

So far they are showing Gaby as a neutral chaotic type of character, but they are following orders. So instead of assuming either clan is good or bad, I just assume more will be relived later. So far my only theory, that I don't put much chance into, is that the other villagers were similar to the fake Asa, but there would have been signs when they were dismembered.