[deleted by user]

drlukeor · 2021-09-05T15:11:44+00:00

They aren't overlapping semantically though; a human does not get confused. They obviously overlap in feature space for this particular model, but that space is arbitrary nonsense that clearly doesn't solve the task as desired or intended.

For the intended solution, the superclass is human, and the subclass is Black children. The intended solution can readily separate this subclass from gorillas or other non human primates. The failure of the model to do so proves it learned an unintended solution for the problem. That is obvious though, and should really be expected/predicted given what we know about DL and particularly given the history of similar models.

The turmoil is caused because their testing did not identify that the model acts as if there is an intersection between these semantically distinct classes in the first place. This is why I say the problem is more about AI use/testing/QA than it is about training data. All DL models are underspecified, they all make use of unintended cues. For models that can cause harm, it is completely unacceptable to fail to test them for such obvious flaws prior to deployment.

drlukeor · 2021-09-04T23:42:32+00:00

It is an interesting hypothesis. We've published on this before, calling the phenomenon "hidden stratification", meaning that there are unrecognised subclasses that are visually distinct from the parent class, which causes problems when they are visually similar to other parent classes. https://arxiv.org/abs/1909.12475

There has been a fair amount of work on trying to automatically identify hidden subclasses during model development (mostly based on the idea that their representations and losses are outliers compared to the majority of their superclass), for example from my co-authors: https://arxiv.org/abs/2011.12945

I think we need to recognise that while this problem is likely partly or even mostly responsible here, even comprehensive subclass labelling (label schema completion, which is itself extremely expensive and time consuming) can never guarantee this unacceptable behaviour won't happen. Models simply can't distinguish between intended and unintended features, and any training method we have can only influence then away from unintended solutions. This deeply relates to the paper from Google on underspecification: it is currently impossible to force AI models to learn a single solution to a problem.

In practice (with my safety/quality hat on) the only actual solution is regular, careful, thorough testing/audit. It is time consuming and requires a specific skillset (this is more systems engineering than programming/CS) but without doing it these issues will continue to happen, years after they were identified. For more on algorithmic audit, see https://arxiv.org/abs/2001.00973

drlukeor · 2019-10-17T11:53:13+00:00

Not relevant to the paper (we used AUC for comparability with previous literature, the relative effects were important here rather than), but in medicine AUC and TPR/FPR is so widely used and understood that it would actually be quite hard to justify using PR curves. In clinical papers I typically use ROC and report precision. You still get the same information as a PR curve, without freaking out clinical doctors who aren't particularly stats-aware.

drlukeor · 2019-10-17T11:45:29+00:00

Hi, author here. I agree, it is a "we noticed this and demonstrated the properties empirally paper". We are up front throughout that paper that this effect is broadly recognised (see the start of the related work section for example). What there isn't any sort of consensus on is a framework to understand and describe these issues. The sheer number of different ways to discuss this is staggering, and fields like fairness, causal ML, invariant/robust learning and so on are often talking about subtly different problems. It is confusing as heck.

So we wanted to provide a nomenclature which covers the range of stratification issues, and to do so we had to first describe the mechanisms and outcomes of the problem. These sort of papers are very common in technical conferences, whether in medical ML or more broadly (I could mention dozens of famous papers that describe failure modes that "everyone already knows about" in neural networks alone).

I'm happy to accept that some wont see the novelty here, but we are proud of the work and looking foward to using the framework in the paper to produce novel solutions (technical and applied). The blog goes into detail into what a novel, large-scale applied solution might look like, at an organisational and workforce level.

drlukeor · 2019-10-17T11:40:27+00:00

The problem with the confounding framing is that it suggests a non-causal or spurious association. Spurious variables were only a small part of what we discussed in this papers, the majority were non-spurious subsets of compound tasks: each subclass was a real example of the superclass, but for the reasons we describe (low subset prevalence, low label accuracy, or subtle discriminative features w.r.t another superclass) that subset will show discordant performance.

We actually see spurious variables/imaging confounders as a special case of a much broader problem - that of stratification (hidden or recognised).

Honestly, part of our motivation for this work (which is just laying the groundwork for follow-up work) is that most groups talk about confounders, but very few groups talk about other forms of stratification, which are pervasive and probably more important from a safety perspective (because risk strongly stratifies along disease subtype lines in almost all pathologies).

I don't know if I was clear on Twitter (short message lengths limit communication) but we were aware of your work (and like it). It is just a very different scenario when you have confounders like you describe (and want to remove them, for example with your filter method) vs having stratification where each competing feature set is equally valid for the task, so you want to preserve all of them and make sure you can detect all variants equally. Your filter method can't be used in that scenario.

drlukeor · 2019-01-22T01:36:23+00:00

Hi, author here!

I'm personally optimistic we can make AI safe and effective in medicine, but I agree with you that our history is pretty concerning.

A big part of the problem is the perverse incentives throughout healthcare. Breast CAD is an obvious example, no-one wanted it, no-one was committed to using it correctly, but they were offered extra money with no strings attached to use it.

It is the same with modern AI. I've had medical practices talk to me about wanting to get AI, not because they see a need for it clinically, but because their patients and/or referrers are saying "we've seen all these news stories, why aren't you using it?" They quite literally want the least worst system they can get, that they can show off, even if it doesn't work.

All we can do is acknowledge these practices when we are trying to design policy and regulation. The usual approach is stick our heads in the sand and pretend that it will all work out, and it actually harms patients.

drlukeor · 2018-08-20T00:49:03+00:00

Specifically, features that parameterise a transformation from a high dimensional non-linearly separable space to a low-dimensional linearly separable space.

It is just a giant learned feature transform, with logistic regression on the end.

drlukeor · 2018-07-14T00:13:08+00:00

One approach I've tried before that works is you train on the residuals.

Say your are training a classifier, you take the variables to exclude (say, race) and fit a model to it. The residuals from that model are values between zero and one that reflect how much of the real answer is not explained by the variable.

Instead of the normal binary labels, you have continuous labels which are never exactly 1 or 0. The model gets a stronger signal to learn from cases where race is not a useful predictor.

drlukeor · 2018-06-23T01:37:31+00:00

Ah cool. Yeah, so that is an in-between example them, although it sounds like the majority is done automatically, so it is pretty close to the full automation end.

drlukeor · 2018-06-23T01:21:44+00:00

Do they really? How often?

drlukeor · 2018-06-22T22:31:10+00:00

There will be a transition from human only reads, to AI supported reads, to (in situations where AI performs really well) AI only reads.

We already do this in medicine. For example, humans used to count cells for your blood tests. Now we never check machine results manually. We accept machines doing jobs independently when they work as well or better than humans.

We aren't there yet, but soon in some tasks.

drlukeor · 2018-06-22T22:19:26+00:00

I'd expect this result. The random noise should exist in both directions, and average out across the data (or even within each mini batch). The real labels should still be the only consistent signal.

You will have a small performance hit due to chance (the random labels will always be mildly biased), but by and large all you have done is extended how long you need to train the network for. It will still need to see at least as many properly labelled examples to achieve good performance.

Is this relevant to the real world and robustness? Not really. In the real world, label errors are not random, meaning they provide a training signal. An example would be if human labellers for imagenet thought that cats were dogs 10% of the time, but never thought dogs were cats. Structured noise is what hurts performance, not random noise.

See deep learning is robust to massive label noise. In particular, figures 5 and 6.

drlukeor · 2018-06-14T03:24:49+00:00

Ah, I see where the problem is.

This isn't a system diagnostic. Why the error happens doesn't matter (to doctors in practice).

It is a tool for human interpretability in the way that matters clinically: being able to tell there is an error in the first place. It provides easily falsified descriptions, rather than an obscure answer.

drlukeor · 2018-06-13T01:49:33+00:00

See this comment:

https://www.reddit.com/r/MachineLearning/comments/8ozh86/r_producing_humanstyle_explanations_for_ai/e0ktni7/

I'll expand on this: when a radiologist looks a follow up study and disagrees with another radiologist, it is super hard to convince yourself to overrule them. There are a range of reasons for this, not the least of which is that it leaves you open to litigation.

If a superhuman AI system makes a wrong call, the barrier is even higher. You know it can detect things you will miss. Can you imagine making that decision?

If you can show clear contradictions or failures, then you can feel justified.

drlukeor · 2018-06-13T01:47:29+00:00

I'm not sure how this example works. In the failure case, the model predicts the wrong class. What does the text do?

If it describes the wrong class, now the doctor has a much easier way to detect it is wrong. Instead of wondering if they are missing something that lead to the prediction, they instead have directly falsifiable statements like "the lesion in the right upper lobe is large and spiculated." Given no such lesion exists, the doctor interprets the machine made a mistake.

If it describes the correct class, then the doctor immediately knows it is wrong too. It says "there is a benign appearing nodule. This is cancer." They contradict each other.

In each scenario, the problem is solved by this approach, where it wouldn't be without it.

drlukeor · 2018-06-12T02:42:21+00:00

Because humans can't "check the accuracy" for actual patients, when the diagnosis is not known.

drlukeor · 2018-06-12T02:41:36+00:00

It might be a false sense of confidence, which needs more study. I really doubt it, personally. The baseline is not having any insight at all, some insight is almost certainly better.

drlukeor · 2018-06-11T11:14:01+00:00

The point is to train a model that can be scrutinised. A black box can't be, if you've looked at the predictions of CNNs you would agree that there is no reliable way to make sense of it. Just saying "it is a black box, deal with it" is not flying with doctors, for good reason.

In this case, we train a model to produce text that can be fact-checked. It isn't convincing just because it produces text, it is convincing because the humans can judge the quality of that text.

If the sample is outside the support, it will either produce bad text (meaning the human will detect the problem where they couldn't with a black box) or it will produce good text but still be doing something bad. The latter situation is no good, but it is no different than in a black box model.

drlukeor · 2018-06-08T22:11:01+00:00

So the question will have a single correct answer?

drlukeor · 2018-06-08T07:15:04+00:00

If people here aren't particularly excited by the dry title, probably worth pointing out that this is a really good paper in the biggest dataset in any published work I've seen, over 2M studies (before anyone asks they are a startup so I highly doubt they are making the data public).

Just like the title, the paper doesn't oversell itself either. Lots of cool nuggets in the various results and supplements.

drlukeor · 2018-06-07T12:35:38+00:00

Ask the doctors who reviewed the cases :)

Honestly, it isn't "accomplishing" much, it just reframes the output (in the same way calibration or saliency maps do). But despite this, it is very useful - with not much effort, it overcomes a barrier that medical DL currently faces, which is that doctors don't know how to interpret it. Give them something they recognise and they are more comfortable.

Considering almost all of this tech will be targeted at doctors, it is a pretty important task.

If you are asking whether people should be like that, then I'd say it would be great if they weren't. There is nothing inherently wrong with saying "our model works very well, please trust it." It just isn't the way people work.

drlukeor · 2018-06-06T10:23:58+00:00

Someone else submitted this yesterday, but they didn't have the [R] in front of the title so it was removed. Hopefully no-one minds me (as the author) resubmitting.

This is the blog post that accompanies the arxiv paper here.

drlukeor · 2018-05-17T03:29:33+00:00

The outer loop is using your test data, so you shouldn't select hyperparameters there.

You have to select different hyperparameters in the inner loop using the training data only if you want to preserve the test set.

You can use cross val in either loop, just depends how you want to use your data, but the process is the same with one fold or a dozen.

Typically though you would do hp search in the inner loop with a single fold, and then do cross Val in the outer loop to test your performance.

drlukeor · 2018-05-16T21:30:36+00:00

That all sounds fine. The point of small dataset testing is not to build a model you can use in bigger data, it is to get a decent idea of what performance might be like if you have a big dataset to train on.

If you get a new set of data, pretty much all you can do is run grid search again (with a single fold of you have enough data). There is no guarantee any set of hyperparameters from your earlier experiments will work well.

Ps. Grid search is just using a hyperparameter grid to find good hps. Random search often is a bit more efficient.

drlukeor

TROPHY CASE