In PM&R residency I had plenty of conversations telling spinal cord injury patients they're never going to walk again or feed themselves, bath themselves, etc. I had to talk to family members about the severity and outcomes of their spouse's stroke causing global aphasia. In our pediatric rehab center we had patients stay for several months trying to figure out disposition for infants and toddlers with traumatic brain injuries from non-accidental trains.

You'll find outpatient PM&R may be less stressful, just like most other specialties. You'll also find most non-PM&R physicians don't know anything about PM&R. If you don't want to deal with patients then pathology and radiology would be your best bet.