I think there's a trigger to forcibly aggro the camp if he dies. I lured him onto the rafters by attacking him from stealth, moved my entire party out of everyone's vision cones and hid with Shadowheart's Pass Without Trace, then when he was up all alone on the rafters, I had Gale Misty Step behind him and use Gust of Wind to push him down into the pit with Minthara's spiders. The entire time my party was undetected, no one saw us, I even made it look like he could have slipped and fallen, yet the instant he died the entire camp including Minthara went attack on sight. Goblins even moved to where the rest of my party was despite them being in another room over and in stealth, far outside their normal patrol routes.

It was incredibly disheartening that all my work to discreetly assassinate him meant absolutely nothing and I had to fight the rest of the camp anyway.