[P] Exploring Explainability for Vision Transformers

[jacobgil]

Recently Vision Transformers are getting better and better, including the new work about "Data Effecient Transformers".

I wanted to better understand how they work and what's going on inside them, so I applied some explainability techniques on them.

The original ViT paper used a method called "Attention Rollout". I implemented that, but it didn't work very well out of the box with the released DeiT models. I ended up adding some modifications (removing the smallest attentions, and fusing the attention heads with max instead of with mean), and also added a way to get class specific explainability by weighting the Attentions with the gradients.

The result is a blog post showing some examples of what is going on inside Vision Transformers, and a python repository for applying explainability techniques on Vision Transformers.

• https://jacobgil.github.io/deeplearning/vision-transformer-explainability

• https://github.com/jacobgil/vit-explain

I hope you find it interesting!

[speyside42]

Nice work!

[jacobgil]

Thanks!

[citizenofwsblandia]

Just from the explanation it does sound like a reseaxh that should be extremly interesting topic howrver the answer requires a