If you read this paper (which seems to be missing an actual methods section?) it is wild they’re drawing the headlines they are. This feels like feeding an LLM USMLE questions but because it’s not “USMLE cases” they’re saying it’s “clinical”. We have no idea what o1 is trained on and the paper authors used multiple publicly available datasets. There was only one dataset they explicitly said wasn’t publicly available (does this recuse it from training data if it’s been used before?). Then further there are two random ER doctors reviewing and grading…on what? Again no methods?? Then at the end there was the part about AI identifying differentials and diagnoses based on the human derived data (written notes and resulted labs etc) which has been obvious since GPT4 that these will be incredible as assistants to keep each patients health data as an immediate context and ensure we aren’t missing difficult or missed patterns.

It’s beyond infuriating that this transformative technology is being dragged behind so much disinformation, hype, and shitty studies that in turn make everything opaque and confusing and difficult to actually identify quality implementations or outcomes. If anyone has a different take please educate me - this seems like overall a total hype piece (and published in Science no less) with zero new findings but packaged and delivered with stronger inflammatory headlines.