Abstract:Relative Positional Encoding (RPE), which encodes the relative distance between any pair of tokens, is one of the most successful modifications to the original Transformer. As far as we know, theoretical understanding of the RPE-based Transformers is largely unexplored. In this work, we mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. One may naturally assume the answer is in the affirmative -- RPE-based Transformers are universal function approximators. However, we present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. One key reason lies in that most RPEs are placed in the softmax attention that always generates a right stochastic matrix. This restricts the network from capturing positional information in the RPEs and limits its capacity. To overcome the problem and make the model more powerful, we first present sufficient conditions for RPE-based Transformers to achieve universal function approximation. With the theoretical guidance, we develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions. Therefore, the corresponding URPE-based Transformers become universal function approximators. Extensive experiments covering typical architectures and tasks demonstrate that our model is parameter-efficient and can achieve superior performance to strong baselines in a wide range of applications. The code will be made publicly available at <a class="link-external link-https" href="https://github.com/lsj2408/URPE" rel="external noopener nofollow">this https URL</a>.

Universal Regular Conditional Distributions via Probabilistic Transformers

Transformers are Universal In-context Learners

On Statistical Rates of Conditional Diffusion Transformers: Approximation, Estimation and Minimax Optimality

Estimating Probability Densities with Transformer and Denoising Diffusion

Your Transformer May Not be as Powerful as You Expect

Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation

Towards Understanding the Universality of Transformers for Next-Token Prediction

Generative Conditional Distributions by Neural (Entropic) Optimal Transport

Latte: Latent Attention for Linear Time Transformers

Estimating Conditional Distributions with Neural Networks using R package deeptrafo

Transformers Can Do Bayesian Inference

Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks

Universal Approximation of Operators with Transformers and Neural Integral Operators

Wasserstein Generative Learning of Conditional Distribution

Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification

Transformer-Modulated Diffusion Models for Probabilistic Multivariate Time Series Forecasting

Universal Approximation of Linear Time-Invariant (LTI) Systems through RNNs: Power of Randomness in Reservoir Computing

Transformers can optimally learn regression mixture models

Towards Identifiable Unsupervised Domain Translation: A Diversified Distribution Matching Approach

Out-of-distribution generalization via composition: a lens through induction heads in Transformers

Teaching Probabilistic Logical Reasoning to Transformers