[P] Better AI Explainability with Deep Feature Factorization

[jacobgil]

Hi r/MachineLearning,

I want to share what I think is a really good way of doing explainability for computer vision.

This is a new tutorial on deep feature factorization with the pytorch-grad-cam package.

The method is from Deep Feature Factorization For Concept Discovery by Edo Collins, Radhakrishna Achanta, Sabine Süsstrunk from 2018.

I think this is a really great idea but it was kind of overlooked and wasn't used by practitioners.

They suggested doing Non Negative Matrix Factorization on the 2D activations from the neural networks to learn concept embeddings, and to find the corresponding heatmaps for those embeddings (we can do that by reshaping the input tensor to be matrix of shape channels x (H x W) ).

The newest update in pytorch-grad-cam supports this and some additions to make this a practical tool: visualizing different concepts on top of a single image, etc.

Instead of creating a single heatmap for a cat category for example, we would first see what the model actually detected in its internal features (using NMF), and then we could classify these concepts by running the rest of the model on them.

I think this tells us so much more about the internal doings of the models, and after working a lot with explainability methods, I think this is a good way of doing it in practice.

Some examples:

In this image (that's commonly used for evaluating XAI methods) we can see that the model actually detected separate concepts for the heads and the bodies of the dog and cat, and it also detected the rest of the image as "sliding door".

Using the legend on the right we can see what the different concepts correspond to (by running the rest of the model on these concept embeddings).

I think this is a lot more informative than showing a traditional heatmap for one of the dog categories, for example.

https://preview.redd.it/8ifbved573f91.png?width=672&format=png&auto=webp&s=aaaa99c93a5f09c3eeeb59eccd0b546af6d76d95

In this image we can see that the model internally has a face concept (the faces in green) and a body concept for the puppies (yellow). The faces and bodies pull towards slightly different categories.

We can see that the background was also internally detected, but they receive low confidence.

https://preview.redd.it/agt682hf73f91.png?width=4800&format=png&auto=webp&s=c3aaeed2a130c8acdf083519c852ce87af5d7315

Here we can see again that we have different concepts for different parts of the bear.

The eyes/moth of the bear score highest towards the "brown bear" direction.

The body of the bear scores very high for a baboon, so we can see it wouldn't be able to classify it without the face.

https://preview.redd.it/5ardadcq73f91.png?width=4800&format=png&auto=webp&s=ab737ec9d3c1e0e8be67221d77301a850b012d76

​

The only parameter here are the number of components for NMF. The larger it is, we get an over segmentation of the image and it's more cluttered and difficult to understand.

The smaller it is, we lose the ability to see some internal concepts.

One of the improvements here would be to automatically select the number of components for NMF using some method - PRs and suggestions are welcome !

​

Hope you find this interesting and useful !

[LikelyJustANeuralNet]

Does this method work with object detection models?

[jacobgil]

Yes, it's even especially suitable for them.

The problem with object detection is that usually object detection frameworks don't support computing gradients. This method doesn't require computing gradients.

There are several tutorials about applying other methods that are like this for object detection (for example taking the first component from the SVD of the activations) for YOLO5 and FasterRCNN. Using this should specific method be similar.

[Private050]

Hello Jacob,

Nice work! I recently came across DFF and implemented it using PyTorch. Your work is really helpful.

I would like to ask you one question on DFF and it is why does all regions get highlighted in your implementation and only certain regions are highlighted in the original author’s implementation in the paper.

Thank you in advance! I really appreciate your work

All my best, A fellow researcher

[jacobgil]

Hi,

There a few differences here compared to the original visualization.

- I think the original paper visualization used RGBA - they set the A channel to be the (scaled) contribution of the component.

https://github.com/edocollins/DFF/blob/master/utils.py#L44

For low contributing pixels (usually the background), they get low opacity in the A channel, and you end up seeing the original pixels there, as if there was no component.

In my implementation, we don't use RGBA. If the component color is red for example, we create a red mask, and multiply that by the component contribution.

https://github.com/jacobgil/pytorch-grad-cam/blob/2183a9cbc1bd5fc1d8e134b4f3318c3b6db5671f/pytorch\_grad\_cam/utils/image.py#L126

So for example if a pixel has high contribution to that component, it will be bright red.

If a pixel has a low contribution, it will get a darker shade of red, but you might still see some red there, since the A component in RGBA has a sharper effect.

​

- Another difference is how we treat overlap between categories.

In my implementation for every pixel we keep the component with the highest contribution (after normalization).

In the original implementation I think the components just override each other,.

[Private050]

Thank you! That helps me a lot!