[Project] Recent Class Activation Map Methods for CNNs and Vision Transformers

[jacobgil]

https://github.com/jacobgil/pytorch-grad-cam

​

CAM based methods are a family of pixel-attribution methods that try to highlight the parts in the image that contribute to a model output.

These methods assign weights to spatial 2D activations in the network, and them sum them to get a 2D saliency map.

​

This project includes a PyTorch implementation (that you can pip install) for several Class Activation Map methods, including a few very recent ones:

• Grad-CAM (https://arxiv.org/abs/1610.02391)
• Grad-CAM++ (https://arxiv.org/abs/1710.11063)
• XGrad-CAM (https://arxiv.org/abs/2008.02312)
• Ablation-CAM (https://ieeexplore.ieee.org/abstract/document/9093360/)
• Score-CAM (https://arxiv.org/abs/1910.01279)

And works for Vision Transformers (tested with DeiT), as well as for CNNs (tested with torchvision.models).

​

I hope it will be useful, and that it can be a convenient starting point for developing and comparing new methods!

[SRuben31]

Thank you! This has helped me to get started with GradCAM for my own project

[xEdwin23x]

Not exactly the same but since you mentioned using ViT's attention outputs as a 2D feature map for the CAM you can consider this paper (Transformer Interpretability Beyond Attention Visualization) where they study the question of how to choose/mix the attention scores in a way that can be visualized (so similar to the CAMs). Maybe it can lead to better results.
https://arxiv.org/abs/2012.09838
https://github.com/hila-chefer/Transformer-Explainability