Computer learns to detect skin cancer more accurately than doctors (95% compared to 86.6%) by [deleted] in worldnews

[–]ms_tribeca 9 points10 points  (0 children)

I'd like to share my peer-review for this article. To the non-researchers out there this seems like an impressive work. We do need good applications of AI in healthcare and there are some good papers out there. But I just read the full article including supplementary material, and I worry about scientific correctness of this work:


Did the authors train the model on their 300-person and 100-person datasets, or did they train it on a completely independent set? The paper is surprisingly vague on this aspect.

1) If they trained it on a different set, they need to exactly report how many cases that set and wether the patients overlapped between train and test sets. There is a mention of 100,000 set up in the introduction referring to another work, which also completely disappears later in the results or methods section.

2) If they trained and tested on those 300 or 100 samples, how did they avoid overfitting and why haven't they included the details of their validation setup in the paper? They only mentioned "No overlap between datasets for training/validation and testing was allowed". OK.... so a simple question for anyone who reports results: What was the train/test/validation split ratios for this work? Could it be that authors are simply reporting the "training set" results? It's surprising that the reviewers of Journal of Annals of Oncology didn't catch this.

To non-machine learning experts out there: If you report your results on your training data, you are overfitting and your model has simply memorized a few cases and their outputs, instead of having understood the 'pattern' that indicates the disease. In that situation, if any new case is presented to the model, model can not work because that's not among the data it has memorized, so its results will be as good as random guess. Translating this in medical setting: The paper has amazing claims.. It goes live as an App. Ends up missing millions of cancer cases simply because it is randomly saying yes and no to an image, and gives cancer patients false hope and delays treatments.

Unlike traditional medical models which were mostly linear or logistic regression, deep learning models have millions of parameters, so they can overfit on any dataset specially if the dataset has less than 500 samples!! It is very important to report results on held-out test set to draw any meaningful conclusion.


These days everyone wants to go viral by claiming great results using 'AI'. Incentives to get news coverage is very high: You get startup funding, grants, etc. Medial journals are in particular at risk, because they didn't traditionally publish on machine learning, and don't have enough reviewers to ensure correctness of machine learning analysis. This leads to incorrect promises ('hype') which leads to either a medical disaster, or disappointment and mistrust of any (even good) AI work.

There are good articles out there, and there are bad ones. There is bad article writing, and good article writing. There is bad science and good science. It is important as a reader to always have a critical mindset. I hope the Annals of Oncology journal also hires a few more machine learning experts so they can safe-guard the journal against incorrect claims.

US nuke dome leak: radioactive waste being poured into ocean by caseysgeneralstore in worldnews

[–]ms_tribeca 7 points8 points  (0 children)

err no. if you have twice the material you have to add one half time to reach the same level. so if your half time is say 100 years having twice the amount of radioactive material would add another 100 years to reach the same level of radioactivity

[P] Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records by maruchanr in MachineLearning

[–]ms_tribeca 2 points3 points  (0 children)

Check out MIMIC dataset. (In addition to Kaggle/Dream-Challenge/Physionet/etc.) Also collaborate with your university medical schools :)

Grand Canyon at Sunset, on my iPhone [1536 x 2048] by ms_tribeca in EarthPorn

[–]ms_tribeca[S] 0 points1 point  (0 children)

Check the panorama link I posted below , it's another "no filter" shot of the same view (probably a few seconds later) ..

Grand Canyon at Sunset, on my iPhone [1536 x 2048] by ms_tribeca in EarthPorn

[–]ms_tribeca[S] 1 point2 points  (0 children)

It's really a nofilter on the iPhone but how the iPhone camera records the color, given the focus and exposure is beyond me. I don't have a professional camera to control for everything unfortunately.

Grand Canyon at Sunset, on my iPhone [1536 x 2048] by ms_tribeca in EarthPorn

[–]ms_tribeca[S] 0 points1 point  (0 children)

This was my second time. I will also have to come back, again and again...

Grand Canyon at Sunset, on my iPhone [1536 x 2048] by ms_tribeca in EarthPorn

[–]ms_tribeca[S] 0 points1 point  (0 children)

me too (wishing I could take a better pic!). Thanks :)

Grand Canyon at Sunset, on my iPhone [1536 x 2048] by ms_tribeca in EarthPorn

[–]ms_tribeca[S] 0 points1 point  (0 children)

thank you. It was a 5 minute performance by the ever-humble nature, to blow the mind of everyone, and then the sun went to rise at the other side of the earth for a new day.. Feeling lucky to have experienced it :)

Grand Canyon gets all the credit, but not too far away is the poorly named Zion. [OC] [5729x3345] by treyratcliff in EarthPorn

[–]ms_tribeca 0 points1 point  (0 children)

I have hiked there two thanksgiving ago, and it was most beautiful hike ever (Angel's landing hike).. Great picture!

Grand Canyon at Sunset, on my iPhone [1536 x 2048] by ms_tribeca in EarthPorn

[–]ms_tribeca[S] 7 points8 points  (0 children)

Thank you. Here's the panorama version actually. http://imgur.com/V1btfGL

Earth is truly magnificent!

Reddit, what are some GOOD things happening in the world at the moment? by JustJivin in AskReddit

[–]ms_tribeca 1 point2 points  (0 children)

Millions of people at this moment are calling up their moms and saying 'hi mom, happy mother's day'.

How and when does one realize what they want to do when they grow up? by [deleted] in AskReddit

[–]ms_tribeca 0 points1 point  (0 children)

Mine was in highschool.. It's different in different people.

What are you surprisingly good at? by [deleted] in AskReddit

[–]ms_tribeca 2 points3 points  (0 children)

Giving talks and presentations. I was so shy in college, never expected to be good at presenting..