Transformer Interpretability from Perspective of Attention and Gradient
Authors: Yongjin Cui, Xiaohui Fan, Huajun Chen
Summary
The authors propose a method to interpret Vision Transformers (ViT) by guiding gradient flow in the direction of attention, which they argue provides more comprehensive and detailed feature-region interpretation. They show that by leveraging the difference between how ViT and humans perceive images, they can alter an image's predicted class in ways nearly imperceptible to humans ("class rewriting"), raising potential security concerns. The paper combines gradient-based and attention-based interpretability to better understand ViT mechanisms.
Main takeaways:
- Standard gradient-based interpretation for Transformers can be improved by explicitly guiding gradients along attention directions.
- This attention-guided gradient method offers more detailed interpretation of which image regions contribute to predictions.
- The method reveals that ViT perceives images differently from humans, enabling "class rewriting" — changing the predicted class with imperceptible image modifications.
- Class rewriting poses potential security risks in deployment scenarios.
- The work aims to deepen understanding of Transformer mechanisms through the interaction of attention and gradient.
Relevance
Tangential to my work on activation steering and fine-tuning — this is about interpreting vision Transformers via attention and gradients, not about language model personas or behavioral installation. Only relevant if I ever need interpretability methods that combine attention and gradients to understand steering mechanisms.
Abstract
arXiv:2605.11392v1 Announce Type: new Abstract: Although researchers' attention is more focused on the performance of Transformer models, the interpretation of Transformer can never be ignored. Gradient is widely utilized in Transformer interpretation. From the perspective of attention and gradient, we conduct an in-depth study of Transformer interpretation and propose a method to achieve it by guiding the gradient direction, or more precisely, the attention direction. The method enables more comprehensive interpretation of feature regions, offers detail interpretation, and helps to better understand Transformer mechanism. Leveraging the difference in how Vision Transformer (ViT) and humans perceive images, we alter the class of an image in a way that is almost imperceptible to the human eye. This class rewriting phenomenon may potentially pose security risks in certain scenarios.