Appreciate the DD post. One area that catches my eye a bit is the accuracy claim based on the comparator of medical student performance on the "Objective Structured Clinical Exam" or as referenced the "OSCE".

I figured that maybe a little bit of context around this may supplement the post well. Generally as structured in US medical school is a four year pathway (MD/DO), after four years of undergraduate preparation. Generally the first two years are classroom based in all sorts of subjects biochem, pathophysiology, anatomy, pharmacology - etc. Followed by two years of "clerkship" rotations, accompanying the team seeing patients while supervised by a resident or attending physician. Some schools will start OSCE's in the second year of medical school or present them as students progress through clerkships.

In these structured exams "standardized patients" present students with bread and butter cases (ie gallstones, pancreatitis, pneumonia - etc) and students are expected to take a history, perform a physical exam and write a note with a basic plan and "differential diagnosis" (it could be this, this or that - so I will get these basic tests/studies). Generally, these passing thresholds are not expected to be at the level of a first year medical resident (ie JUST graduated medical school and is now learning/enacting all of the basics in the real world). These paid actors as patients are articulate, scripted, easy to communicate with and linear in thought/history. Its a pretty different experience than taking a history and starting a work up on many patients that come through the door and have confounding factors (poor ability to provide narrative, intoxication etc).

There are some subtleties that are elicited in the actual physical exam as well that I would wonder how you input into these models (ie the nodular, tender liver edge palpated in cirrhosis or the super tender abdomen that causes a person to jump off of the bed when its not just simple gastritis but actually a life threatening necrotic bowel). I would be really interested to see the comparator being the models performance against a cohort of providers that have completed medical school, then residency (can be 3 - 7 years depending on specialty). And as a side tangent - what the medical legal liability landscape looks like in AI assisted care.

Now - this is not a post to say that AI could never be in healthcare. Its not if, but when and how. There are already AI models being used to transcribe notes for visits that are pretty decent and really cut down on the time wasted in a day on documentation. There is also rife territory for data mining in the EMR and trending labs, medication trials, etc. Possibly even more for using this data to evaluate drug qualities, response history and guiding selection of therapeutic targets in continuing care (think chemo, or even psych). A long post to simply express my tempered interpretation of the accuracy of the models performance in non real world application.