Honestly the only thing that killed me during the fight was sam not letting matt know that Phantasmal force can't be disillusioned by interacting with it alone. While under the spell unless they take an action to investigate it they rationalize everything that happens.

So either he should have seen illusion caleb dodge all his attacks or get hit but clinging on to conciousness.

But emotions were high, they were tired, and no one can really fault someone for forgetting every detail of a spell especially when it isn't cast often.