Abstract:Since its inception in "Attention Is All You Need", transformer architecture has led to revolutionary advancements in NLP. The attention layer within the transformer admits a sequence of input tokens $X$ and makes them interact through pairwise similarities computed as softmax$(XQK^\top X^\top)$, where $(K,Q)$ are the trainable key-query parameters. In this work, we establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem that separates optimal input tokens from non-optimal tokens using linear constraints on the outer-products of token pairs. This formalism allows us to characterize the implicit bias of 1-layer transformers optimized with gradient descent: (1) Optimizing the attention layer with vanishing regularization, parameterized by $(K,Q)$, converges in direction to an SVM solution minimizing the nuclear norm of the combined parameter $W=KQ^\top$. Instead, directly parameterizing by $W$ minimizes a Frobenius norm objective. We characterize this convergence, highlighting that it can occur toward locally-optimal directions rather than global ones. (2) Complementing this, we prove the local/global directional convergence of gradient descent under suitable geometric conditions. Importantly, we show that over-parameterization catalyzes global convergence by ensuring the feasibility of the SVM problem and by guaranteeing a benign optimization landscape devoid of stationary points. (3) While our theory applies primarily to linear prediction heads, we propose a more general SVM equivalence that predicts the implicit bias with nonlinear heads. Our findings are applicable to arbitrary datasets and their validity is verified via experiments. We also introduce several open problems and research directions. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.

Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation

STRec: Sparse Transformer for Sequential Recommendations.

Towards Efficient and Effective Transformers for Sequential Recommendation.

An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

SSE-PT: Sequential Recommendation Via Personalized Transformer

Scaling Sequential Recommendation Models with Transformers

Attention Calibration for Transformer-based Sequential Recommendation

Sequential recommendation: A study on transformers, nearest neighbors and sampled metrics

MaTrRec: Uniting Mamba and Transformer for Sequential Recommendation

Non-Stationary Transformer Architecture: A Versatile Framework for Recommendation Systems

Integrating Offline Reinforcement Learning with Transformers for Sequential Recommendation

Optimizing Encoder-Only Transformers for Session-Based Recommendation Systems

Adaptive Disentangled Transformer for Sequential Recommendation

Sequential Recommendation with Bidirectional Chronological Augmentation of Transformer

Sequence and Distance Aware Transformer for Recommendation Systems

Improving Transformer-based Sequential Recommenders through Preference Editing

AdaMCT: Adaptive Mixture of CNN-Transformer for Sequential Recommendation

ConvFormer: Revisiting Transformer for Sequential User Modeling

Transformers for Supervised Online Continual Learning

RLT4Rec: Reinforcement Learning Transformer for User Cold Start and Item Recommendation

Transformers as Support Vector Machines