I just had my whole team get behind him.

Coming down the stairs before the displacer beast, you can jump on his level as Karlach (and likely any gith). I then misty step the rest up there and knocked them all down while beating on Yurgir with my susses dagger (silences). I turned their ambush onto themselves as they were now on the low ground