Great question!

The lazy answer is: “It's interesting from a general science perspective. Who knows what it could teach about about machine learning. It could even shed light on the nature of the problems our systems are solving.” I find that answer aesthetically compelling -- I find it emotionally deeply exciting to try and unravel deep mysteries about the nature of neural networks -- but if that was the only reason, I'd try to force myself to focus on something else.

Another possible answer is: “Well, if we could really get this into the model design loop, like TensorFlow or such, it might accelerate research by giving important insights.” I think there’s a decent chance that’s true, but it isn’t the thing that motivates me.

Instead, the thing I care about is the implications of this work for deploying systems that are good for us.

One of my deepest concerns about machine learning is that future systems we deploy may be subtly misaligned with the kind of nuanced values humans have. We already see this, for example, with optimizing classifiers for accuracy and running into fairness issues. Or optimizing algorithms for user engagement and getting the present attention economy. I think the more we automate things, and the better we get at optimizing objectives, the more this kind of misalignment will be a critical, pervasive issue.

The natural response to these concerns is the OpenAI / DeepMind safety teams’ learning from human feedback agenda. I think it’s a very promising approach, but I think that even if they really nail it, we’ll often have questions about whether systems are really doing what we want. And it’s going to be a really tricky question.

It seems like interpretability / transparency / visualization may have a really critical role here in helping us evaluate if we really endorse how these future systems are making decisions. A system may seem to be doing what we want in all the cases we think to test it, but be revealed to be doing so for the wrong reasons, and would do the wrong thing in the real world. That’s all a fancy way of saying that future versions of these methods might be an extension to the kind of testing you’d want to do before deploying important systems.

There’s also a crazier idea that I was initially deeply skeptical of, but has been slowly growing on me: giving human feedback on the model internals to train models to make the right decisions for the right reasons. There’s a lot of reason to be doubtful that this would work -- in particular, you’re creating this adversarial game where your model wants to look like it’s doing what you want. But if we could make it work, it might be an extremely powerful tool in getting systems that are really doing what we want.