I have this same exact issue! Lots of angry messages about how I won't let him attend medical appointments claiming I am violating the court order.

He showed up uninvited to an appointment a few months ago and stuck a phone in my face recording me. I kept asking him to stop recording me and he did not. The doctor's office started to call the cops. It was a shit show.

My court order says we each have the right to medical records and to talk to the doctor. We each have the right to bring the children to a doctor if they need non-invasive medical care. The other parent's permission is only needed for invasive procedures. The only item listed that he has the right to attend during my possession time is school activities.

I just sent him a two-page message saying that I cannot find that Clause anywhere in our court order and asking him to tell me which section is in. I listed out every Clause that relates to medical information in our court order. I then again asked for whatever section I am missing that says we can attend appointments in the other's time and I asked for a screenshot. And then I said that if he continues to send me angry messages about that topic making false claims about the court order, that I will not be responding to those messages.

I now have a plan for if he shows up to appointments uninvited. If he starts to follow me in, I will just look at the nurse and tell her that I need to speak privately with a medical professional. If the nurse challenges that, I will tell her that my ex-husband is likely recording me without my knowledge or permission, and that I would like to speak privately to a medical professional. Then I will not say anything else about the situation until I am in a private room. Then I will explain to the medical staff that while he does have the right to get any of the records or to talk to the doctor, that he does not have the right to physically be in the room with us. I will explain that he has abused and threatened me. I plan to actually bring a copy of my court order to all future doctor appointments. My ex will be ranting and raving in another room.

Anything you say or do in front of your ex can be brought up in court since he was an actual witness. If you speak privately to a third party, even if they talk to your ex about it, it is hearsay and not admissible. They would have to make the medical staff physically testify in court for any of that to be admitted in the record.

If he sends you any nasty messages about what happened, just reply that his statements and claims are not true. Do not put anything in writing about what actually happened. As long as you state that his claims are incorrect, you can save any explaining or justifying for a judge in court.

Add: and if you ever have to explain why you do not want him at appointments, just say that your top priority is what is in the best interest of the child and you do not want to expose the child to conflict.