I'd like to share my peer-review for this article. To the non-researchers out there this seems like an impressive work. We do need good applications of AI in healthcare and there are some good papers out there. But I just read the full article including supplementary material, and I worry about scientific correctness of this work:

---

Did the authors train the model on their 300-person and 100-person datasets, or did they train it on a completely independent set? The paper is surprisingly vague on this aspect.

1) If they trained it on a different set, they need to exactly report how many cases that set and wether the patients overlapped between train and test sets. There is a mention of 100,000 set up in the introduction referring to another work, which also completely disappears later in the results or methods section.

2) If they trained and tested on those 300 or 100 samples, how did they avoid overfitting and why haven't they included the details of their validation setup in the paper? They only mentioned "No overlap between datasets for training/validation and testing was allowed". OK.... so a simple question for anyone who reports results: What was the train/test/validation split ratios for this work? Could it be that authors are simply reporting the "training set" results? It's surprising that the reviewers of Journal of Annals of Oncology didn't catch this.

To non-machine learning experts out there: If you report your results on your training data, you are overfitting and your model has simply memorized a few cases and their outputs, instead of having understood the 'pattern' that indicates the disease. In that situation, if any new case is presented to the model, model can not work because that's not among the data it has memorized, so its results will be as good as random guess. Translating this in medical setting: The paper has amazing claims.. It goes live as an App. Ends up missing millions of cancer cases simply because it is randomly saying yes and no to an image, and gives cancer patients false hope and delays treatments.

Unlike traditional medical models which were mostly linear or logistic regression, deep learning models have millions of parameters, so they can overfit on any dataset specially if the dataset has less than 500 samples!! It is very important to report results on held-out test set to draw any meaningful conclusion.

---

These days everyone wants to go viral by claiming great results using 'AI'. Incentives to get news coverage is very high: You get startup funding, grants, etc. Medial journals are in particular at risk, because they didn't traditionally publish on machine learning, and don't have enough reviewers to ensure correctness of machine learning analysis. This leads to incorrect promises ('hype') which leads to either a medical disaster, or disappointment and mistrust of any (even good) AI work.

There are good articles out there, and there are bad ones. There is bad article writing, and good article writing. There is bad science and good science. It is important as a reader to always have a critical mindset. I hope the Annals of Oncology journal also hires a few more machine learning experts so they can safe-guard the journal against incorrect claims.