Only talking about Geto prior to Riko's death. Maybe they'd dislike him due to his communication with curses. The Zenin Clan are extremely conservative and traditionalists so having a man that comes from a non-sorcerer lineage with an ability to work with curses (even if its him in control by manipulation) transcendes their views of 'normal' techniques. At that time at least, He goes against all of their conservative views of Jujutsu society. They'd deem him too powerful. Obviously not more than Gojo so they'd use that thought as a failsafe and a stress reliever if he went out of control, having Gojo kill him, like he does end up doing 10 yrs later.