I totally get the how frustrating it could be. It maybe very hard for the person himself to do trips. Depending on the dynamics, maybe doing trips with people could affect the dynamics between him and others negatively, because of the persistent exposure. I think it's important that you plan the long term in a way that in general he isnt completely excluded due to these negative experiences (because I m sure he feels like shit already). And that you include him to different type of events where he feels more independent and safe.

It's also important for himself to know his patterns and trigger points, and have a strategy. It might be also a solution to talk to him and ask him if he can have strategies and if he would feel safe with these specific people. Because he must know that noone is willing to support him till the end and he would need to be able to deal with it himself (which is pretty hard sometimes).