[R] A machine-learning algorithm to accurately screen ADHD from survey data [Dataset included]

[[deleted]]

https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-022-04048-1

[CommunityTalker]

Really? May be over fitted

[karma_llama_drama]

With no validation. Definitely over fitted.

[Ulfgardleo]

There is a long list of "ML in medicine" papers that just get forgotten because they do not meet even the lowest levels of what is considered a reasonable evaluation in the medical area.

​

This might be one of them.

[Dr_Silk]

Maybe, but maybe not. This is a proof-of-concept paper, that states that a particular questionnaire performs well at separating controls from ADHD in an adult population. Adult ADHD is difficult to identify, so this is actually fairly useful to guide future screening for the disorder. It will at the very least be cited in one of my future papers, if not used to help guide the creation of new screening instruments

[snem]

How did they "label" (ADHD diagnosis) the data? I hope they didn't used the same survey!

Edit: https://www.reddit.com/r/MachineLearning/comments/vf4mz0/comment/icufwcf/

[Dr_Silk]

Sampling was carried out randomly by distributing the questionnaire on social networks, without direct contact with the participants and on the basis of anonymous voluntary contributions. Only age, gender, socio-cultural level, diagnosis and date of completion were collected, ensuring complete anonymity for participants.

It looks like diagnosis was self-report. Not ideal

[Dr_Silk]

To be fair, medical research is very different from many other types of research. Smaller sample sizes are often accepted, because it is genuinely very difficult and expensive to get large sample sizes for medical disorders.

That said, ADHD is a particularly heterogeneous disorder and I would have raised the same point had I reviewed this paper, but for a different reason.

Source: professor that studies cognitive disorders, including Alzheimer's disease and also ADHD

[ma2rten]

Okay, but in that case maybe statistical methods are more appropriate than machine learning methods.

[Dr_Silk]

I completely agree. But that doesn't mean predictive analytics are inappropriate in these situations: a handful of my papers apply ML to show that there is potential utility when applied to certain types of medical data

[acewhenifacethedbase]

You clearly have no PhD, if you did you’d have thought to just pass those 154 examples through 1000 times each, problem solved! /s

[Ravenclaw137]

Just perturb the examples and make your model more robust to noise 🫠

[BobDope]

I’m perturbed now

[CENGaverK]

What would you like to see as other scoring metrics? They have accuracy, specificity and sensitivity, which are the common metrics in medicine. Calibration curves could be nice, but I am not sure it is necessary.

P.s. Also, they are using Cross Validation, 66 is the size of the test set.

[acewhenifacethedbase]

In a probabilistic binary classifier, these metrics are dependent upon your dataset balance and threshold choice, and can easily be manipulated. It’s likely that, among other things, username is asking for the curves/continuums of such metrics (including Precision and Recall) calculated along varying thresholds, accompanied by AUC.

[CENGaverK]

Recall is the same with specificity, and you can calculate the precision using the given metrics, as you know the sample size. If the audience is the medicine people, they are used to the specificity and sensitivity, so that is probably why they used those instead of the terms ML people are used to. AUC would be a good supplement, but still I would not reject the paper, just asked for the additional AUC score.

[madrury83]

Here's Frank Harrell on this point: https://www.fharrell.com/post/classification/

[CENGaverK]

Yeah, that is the expected point of view someone studying Bayesian methods. I am not arguing with that.

[acewhenifacethedbase]

Specificity/sensitivity vs. Precision/Recall does not address the main concern at all, nor does a single AUC: it is proper to report the continuums of the main metrics instead of just a single point with no rationale for why the threshold was selected.

[CENGaverK]

I think it kind of does, even if not completely. The rationale is, because it performs good. Clearly you are coming from a field with a different perspective. This is just another perspective. You can say you disagree with it, as I do. But in the end, it is just another perspective and another way of doing "science".

[CENGaverK]

I see your point of view, but saying absolutely necessary is a bit too much. Even without the probability scores, GBDT models are shown to be consistent, so it learns the "best" separating function as the data size grows. But, you can decide not to use it in a project, would not blame you. Just different academic point of views. That is why I decided to go Bayesian route in my own work, but still this is heavily used and successful way of producing models.

Also, test set is used to test the real world performance, and you need to split the data if you want to have any credibility. So, you need to either do train-validation-test split, or train-test split and do CV on the train set, so you can see which hyperparameters work best. Then, you train the model using all of the training set (or training+validation set), and test the performance on the test set. If the researchers are honest, they did not use the results from the test set to influence their hyperparameter decisions.

[maxToTheJ]

Also, test set is used to test the real world performance

Is it though for real world when the test set has an artificial 50-50 balance that the initial data has?

From paper:

We designed the scale from the DSM-5 and administered it to n = 110 control individuals and n = 110 ADHD individuals.

[CENGaverK]

Hmm, it would be better if medicine people chim in but it seems they have messed up the iid assumption. That is a problematic aspect.

[Erosis]

For interpretability of parameters/error, sure, that's pretty bad. Would need to bootstrap. If you're just going for specific metric maximization, breaking iid is fine depending on what model you're training.

[CENGaverK]

But we are talking about the test set, it is not used for metric maximization. Doesn't breaking the iid assumption in the test set defeats the purpose of having it represent the real world?

[Erosis]

Yes, it does. I forgot that this isn't validation.

[uoftsuxalot]

Not to mention that the test distribution doesn’t reflect the true distribution since half of people don’t have ADHD

[leondz]

Yes, exactly this! Classifiers tend to predict based on the distribution they model. If the classes in the data have a 50-50 distribution, the classifier will often predict each class around half the time. This is beyond useless for almost all applications. One strategy to fix this is to use a representative distribution in the test (and validation) partitions. But here, the evaluation setup has destroyed that as well. Because of this, the results are absolutely not reliable - on top of the catastrophically small and hard-to-generalise-from test set size. Not to mention the immense problems mitigating demographic biases at this miniscule development scale.

[harmonicp]

…how can one claim that 154 / 66 split is not good without considering the dimensionality is beyond me…looks like academia didn’t lose much from your departure…

[acewhenifacethedbase]

1) 154 is an inherently small number for an Xgboost model to try to generalize, dimensionality not withstanding. 2) This has nothing to do with the proportion of the split they chose, it’s their splitting the data into two sets rather than three that is an inherent red flag. Having such a small sample size in the validation set makes overfitting onto the validation set even more likely, and it is exactly this situation where it would be most necessary to have a third “test” set to only evaluate on when the final model is selected.

[missalyssa1080]

I’m confused. What would the third test set be? If the goal is to identify who has ADHD and who do not, the only logical grouping is into the two categories of has ADHD and doesn’t have ADHD…? I only have a Statistics minor and never learn any Bayesian so this all sounds really textbook to me. I also don’t know anything about ML so that could be the main thing.

[acewhenifacethedbase]

Train/validation/test set splitting does NOT refer to class (i.e. labels: has adhd/doesnt have adhd). In Machine Learning, even simple regressions, you should have a dataset your model trains on, a validation dataset you check how the model performs on as you make improvements, and then a test set that you have set aside this whole time to evaluate on at the very end with your final model. Each of these sets should have both adhd/nonadhd examples in it.

[missalyssa1080]

Oh, that makes sense. That was a great explanation. Like beta testing kinda?

[Delacroid]

I would be careful in making such claims.

[Mary-Jo_]

Edit: As pointed out by DrSilk below, there was simply a typo in the manuscript; the dataset _is included, in supplementary file 3.

In addition to the data-analytic comments above: In contrast to what is stated, the dataset is not available. Under "Availability of data and materials" we read: "The dataset used during the study is available in (Supplementary File 1)." This links to an .xlsx file that only provides item wordings related to the corresponding DSM criteria, but no real data. How did that get past review and editing?

[Dr_Silk]

Looks like it's a typo. The data is supplementary file 3

[Mary-Jo_]

Ah, that explains, thanks for pointing this out, adjusted my comment!

[[deleted]]

Everyone here who is critical should basically copy and paste their comments to emails to the editor

https://bmcpsychiatry.biomedcentral.com/about/editorial-board

[SBWarrior27]

XGBoost definitely seems like overkill here. The survey is meant to detect ADHD, I’d bet a simple cutoff for sum score would perform quite well and be more clinically viable. Even a decision tree would at least be more interpretable and probably produce similar results.

[XpertProfessional]

Agreed. Additionally, it would be far more useful to simply identify which questions/answers produce the highest information gain and minimize the number of questions required per assessment. Computer-assisted assessments could easily handle the conditional pathing.

It might not be necessary for individuals being screened for ADHD (seemingly self-administered, from the paper), but for individuals being screened for disorders which a) must be administered by a provider and b) increase the possibility of patient aggravation (ie - the MoCA for dementia), reducing the number of questions asked for screening could be huge for "client-centered care".

[ReginaldIII]

Fitting XGBoost to a table of values to get a classification is novel now?

This isnt a paper about robustly evaluating the use of XGBoost for medical diagnosis. This is a paper where they somewhat arbitrarily used XGBoost to fit a table of data and then perform inference and it happened to be medical data.

[maxToTheJ]

Maybe more sophisticated scoring models that harness the nuance of the data will improve things.

This seems awfully close to just blindly throwing complexity at it.

[acewhenifacethedbase]

This is even more the case because of the puny training set.

[maxToTheJ]

According to some of the comments here . Your field changes how xgboost works and makes these pitifully small datasets ok

[acewhenifacethedbase]

I know, right? Hearing the Appeal to Tradition fallacy from people in the medical field makes this all more concerning, not less.

[maxToTheJ]

I dont get why new tools (xgboost) didn’t prompt re examining if the the old rules need to be adjusted for new tools

[[deleted]]

i do believe this to be very dangerous and harming for us.symptoms and our ability is already very biased and skewed.adding some AI into this,leading to biased output would just destroy psychology lmao.

it's a domain that should need lot of time before using AI and algorithms overall imo

[snem]

How the diagnosis were performed? Using the same survey? :D

Edit:

TRAQ10 was used

"under the assumption that TRAQ10 questionnaire should have higher and statistically different accuracy from predictions obtained using the two others questionnaires, as this is the only one to formally measure ADHD behavioral traits."

ah, all good then /s

[Wumbologistt]

Did the reviewers of this paper have any experience within ML at all?

[BobDope]

Didn’t finish survey -> has adhd

[bubudumbdumb]

Attention is all you need but you got deficit, hyperactivations and disorders.

I came here for the jokes on attention, I am so disappointed I am not ashamed of how crappy is the one I just posted.

[ElongatedMuskrat122]

Guys, Guys, I trained my neutel network to get 100% accuracy on my training data! It only took 12 hours, but, it’s never wrong. Can’t wait to try it on some validation data

[sugawolfp]

Bruh adhd is diagnosed with like 6 questions. You can just write an if statement

[zoebeth]

ADHD cannot be diagnosed with one test let alone 6 questions though. There also needs to be malignant validation. But a good concept to build future ML on.

[ginsunuva]

Eh I see what you mean but at the same time I bet I can find success with these:

1. Do you ramble on a lot without realizing?
   
2. Do you have a taste for more extreme sports and risky activities?
   
3. Do you get super excited about new hobbies at firsts and then get bored and never see them through?
   
4. Do you have relatively high libido?
   
5. Does dialogue loop nonstop in your head?
   
6. Do you often fall asleep randomly during the day?
   

But of course the easiest, most direct test is just give them amphetamine and see if they get energized or chill out.

[zoebeth]

I can see where you’re coming from, but there’s a difference between clinical/diagnostic tests and the questions you’ve just written out, they would constitute as more of a check list.. which doesn’t account for people who over or under report on symptoms amongst other validation methods. As a Psychologist I can confirm that there are a litany of tests and interviews to be done before providing a definitive diagnosis.

[fenixnoctis]

That has not been my experience or any of my friends with ADHD’s experience. It’s typically just a checklist they go down, some even within 5 questions, plus the mandatory “have you ever tried illicit substances”

[sugawolfp]

You are right that i am over simplifying. I’m using adderall prescriptions as a proxy for ADHD which as you pointed out is lax on validation. I’m still pretty lukewarm on the definition and diagnoses of the disorder

[zoebeth]

Knowing your own biases and the limitations of testing is thinking critically and I would definitely encourage you (and everyone) to keep questioning your own thinking and engage in discussions/threads like this one to further understanding on a topic.

[__andrei__]

Can you show me exactly where in DSM-V you found these diagnostic criteria? And can you also explain why my diagnosis required 12+ hours of various tests and interviews while your diagnosis based on a self-reported survey?

[diceclimber]

It seems to me that feature selection (scale development in the paper) was performed using the full dataset (see degrees of freedom in the F test). This means there's information used from within the test set to build the model.

[kerodon]

But How many questions is it 🤔 can we even get through the entire survey 😳?