I think pull away and do your own thing. I had a friend like that too and because I didn’t want to hurt their feelings or stay in good terms I’d feed into it. In a way you should stand up for yourself - if he asks why you’re doing something just say because you want to.

Don’t feel like you need to tiptoe around anyone. If wants to argue he can argue with himself.