Abstract:With the attention mechanism, transformers achieve significant empirical successes. Despite the intuitive understanding that transformers perform relational inference over long sequences to produce desirable representations, we lack a rigorous theory on how the attention mechanism achieves it. In particular, several intriguing questions remain open: (a) What makes a desirable representation? (b) How does the attention mechanism infer the desirable representation within the forward pass? (c) How does a pretraining procedure learn to infer the desirable representation through the backward pass? We observe that, as is the case in BERT and ViT, input tokens are often exchangeable since they already include positional encodings. The notion of exchangeability induces a latent variable model that is invariant to input sizes, which enables our theoretical analysis. - To answer (a) on representation, we establish the existence of a sufficient and minimal representation of input tokens. In particular, such a representation instantiates the posterior distribution of the latent variable given input tokens, which plays a central role in predicting output labels and solving downstream tasks. - To answer (b) on inference, we prove that attention with the desired parameter infers the latent posterior up to an approximation error, which is decreasing in input sizes. In detail, we quantify how attention approximates the conditional mean of the value given the key, which characterizes how it performs relational inference over long sequences. - To answer (c) on learning, we prove that both supervised and self-supervised objectives allow empirical risk minimization to learn the desired parameter up to a generalization error, which is independent of input sizes. Particularly, in the self-supervised setting, we identify a condition number that is pivotal to solving downstream tasks.

Attribution Rollout: a New Way to Interpret Visual Transformer

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

AttentionViz: A Global View of Transformer Attention

Better Explain Transformers by Illuminating Important Information

A Survey on Visual Transformer

How Does Attention Work in Vision Transformers? A Visual Analytics Attempt

VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

A Survey of Visual Transformers

Explainability of Vision Transformers: A Comprehensive Review and New Perspectives

STAA: Spatio-Temporal Attention Attribution for Real-Time Interpreting Transformer-based Video Models

Demystify Transformers & Convolutions in Modern Image Deep Networks

Visual Analytics for Generative Transformer Models

Context-Sensitive Visualization of Deep Learning Natural Language Processing Models

Transformers in computational visual media: A survey

Dodrio: Exploring Transformer Models with Interactive Visualization

AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers

An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models

A Multiscale Visualization of Attention in the Transformer Model

Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer