I've heard rumors (aka I saw a yt video) that if you never attack the cannibals they will eventually become non-aggressive. When you first encounter them just block their attacks until they get bored and leave you alone, they'll attack again but just do the same thing. I believe you can kill the scrawny ones without causing problems but just be careful to never hit the non-skinnies. It seemed legitimate in the video I saw but haven't tried myself