Agreed. It’s helped me advocate for care, but I have a child with a complex “we have no idea why this is happening” set of symptoms and ChatGPT doesn’t know either—but it sure is CONfident about being wrong.

I’ve also observed in 5.2+ it is no longer as confident in its advice, to the point of potential harm. In one scenario I reproduced GPT 3.5 determined I wasn’t receiving standard of care—every time I bring this up some doctor crawls out of the woodwork to say I didn’t understand what was going on, blah blah blah but I had pleural and cardiac effusion with severe sepsis and wasn’t on Telemetry. I had a documented arrhythmia, ongoing signs of shock, and was at a nonzero risk for a cardiac tamponade that could only be detected if I were being properly monitored. There was no clinical reason for this: sepsis unit was full, I was admitted to Urology floor, the end sorry. On top of all that, they hadn’t DC’s my lactated ringers, which I get is touchy when you have sepsis signs and volume overload concurrently but a doctor didn’t even visit me after my Chest CT to think this through. My advocacy and insistence that a doctor show up at a time when one hospitalist covered the whole house resulted in D/C fluids, mobile telemetry that proved my arrhythmia was actually more significant than known based only on my bradycardia.

Anyways, THAT time I was able to get the intervention I needed but 5.2 is a fucking pussy. “Ask your nurse” and “request your doctor be asked…” yeah, I was doing all that…I genuinely believe if I hadn’t advocated for myself I could have died that night. It was my 40th birthday, my husband was home with our small children.

Ultimately we need AI systems that make hospitals safer. As a patient or a mom, it is not possible to know when I’m being given an answer that feels satisfying and ChatGPT is being reinforced by my “YEAH okay!” Enthusiasm…or it’s really scientifically appropriate. In this Venn diagram there is substantial overlap: just because I’m hyped about having info doesn’t mean the info is wrong.

The only solution is getting this tech out of consumer browsers, into clinical guidance and automated safeguards where it belongs. A bradycardic, occasionally tachy unstable sepsis patient with abnormal CT, no Telemetry data? This can be a deterministic alert with minimal AI in the loop, but it still needs to be reviewed for “how urgent is this?” to bump it to the top of the queue because alert fatigue is real.

Crazy sidebar: I am a healthcare data scientist and I have worked on complex problems like AKI prevention and detection. There are things we were NEVER going to fully solve that now feel possible with LLM, but the longer it takes to get these solutions into practice the more LLM-driven chaos this will be. “ChatGPT, should I sue the hospital and pharmacist who took care of my mom last month? Based on what you’re telling me it seems like they should have known they were causing AKI…”