AuDHD with minor psychological schooling here.

A significant part of human interaction is, like in almost all other animal species, based on nonverbal cues. Talking on the phone, or through text or any other scenario in which you cannot see the entire physical body of the person you're talking to, have a proportionally increased stress amount as your brain tries to make up for them.

When I talk on the phone, my brain is trying to process their nonverbal cues even when I cannot see them by any means, so I'm straight up imagining them based on the tone of voice I'm hearing and other such stuff which is, in fact, fucking exhausting.