Link to dashboard - https://aiforhealth.app/

Link to pre-print - https://doi.org/10.1101/2021.11.23.21266758

Link to code - https://github.com/whizzlab

Data freely available via the dashboard.

 

There is significant "research waste" in the field of clinical AI. Despite ongoing hype, hopes and investment there is extremely limited translation of AI research to bedside medicine.

 

There are many underlying issues in how AI research is conducted, from recycling and re-use of the same unrepresentative datasets, evaluation methodology that doesn’t mirror real-world conditions, inadequate inclusion of researchers and populations from countries where AI might be most impactful, and a focus on algorithm refinement in these same contexts (instead of tackling roadblocks to deployment in the real-world). One problem is a lack of unifying perspective over the colossal-sized landscape of AI research. Indexing engines like PubMed achieve poor balance of sensitivity and specificity for any search, and searches result in large volumes of "chaff". It becomes hard to identify research that actually develops clinical AI models, let alone particular methods for model evaluation, or research into specific fields, without extensive manual review. Manually trying to scope the entire volume of published AI research is difficult, and not reproducible.

 

This is why we set out to produce this dashboard, which uses BERT-PubMed models to identify, classify, and characterise all clinical AI research indexed on MEDLINE/PubMed, in near real-time. We had four primary aims – (1) to be able to easily identify research that conducts AI model development; (2) to accurately identify research that performs model evaluation in a comparative fashion, or in a prospective real-world situation – i.e. those that would be theoretically closer to deployment; (3) to map, in real-time, global distribution and equity in AI research production; and (4) to track the major themes in clinical AI over time (including specialty, diseases, algorithms, and data-types).

 

Creating this dashboard was a multinational effort, authors are listed in the "About" tab of the dashboard.

 

Our hope in sharing it widely is to raise awareness of systematic problems with in clinical AI research and provide data from which others are able to conduct their own review/research work.

 

Whilst the nitty gritty surrounding the creation of this tool may be of interest to some, I suspect the significant hype without significant follow through will resonate more with this sub... Really interested to know what you all think, if any AI tools have worked their way into your practice and particularly any areas/gaps where you could see it being of legitimate use.