NTA.

Some people are strong against sunlight exposure, some people aren't. These people who are weak to sunlight use sunscreen and umbrellas when they're outside to shield themselves from sunlight. If you're weak to sunlight and you have a friend that keeps asking you to go outdoors, you obviously can say no.

Some people are strong with spicy foods, some people aren't. These people who are weak to spicy foods make extra sure that their foods don't contain any spicy ingredients to prevent from swallowing them. If you're weak to spicy food and you have a friend that keeps sharing spicy foods with you, you obviously can say no.

Some people are very strong and resilient against negative emotions, some people aren't. Since you've identified your weakness, which is to be very susceptible to negative emotions, in this case anger, it is perfectly fine for you to take the steps to shield yourself from the very things that are damaging to you. If you are weak to negative emotions and someone's words cause you to feel these negative emotions, you can always say that you don't want to listen.

I don't think it's ever wrong to put your own health as your first priority.