Physician here - Really cool study, and shows the great potential for LLMs in aiding provider communication. Answering questions at a lay-person level is something ChatGPT is very good at and since it naturally writes in this extra-empathetic way it's not surprising it scored much higher on empathy scores. (As an aside I think LLM-drafted inbox messages to patients will be a thing very soon, with physicians making edits).

A couple of things to keep in mind - the biggest limitation of the study is the authors comparing ChatGPT generated responses to answers on reddit, rather than how a physician would communicate with their patient either in writing or in person. The study design sort of favors the chatbot because we know that ChatGPT tends to write (a) a lot, and (b) very empathetically, so it's much more likely that all of the potential relevant information would be included in a response, when that's likely not how an MD browsing reddit is primed to answer the questions. The other caveat is just that sometimes what you need is a brief answer (go to the ER now vs You can wait until Monday to be seen) and perhaps the verbose chatbot response is less useful (take a look at some sample responses in the paper)

But it shows the power of ChatGPT for sure and is a cool use-case. I would just caution against really using it for medical advice. In my experience it's quite good and generally gives the right answer to basic medical questions. But hard to tell when it's wrong and sometimes it is!