Abstract:When the predicted sequence length exceeds the length seen during training, the transformer's inference accuracy diminishes. Existing relative position encoding methods, such as those based on the ALiBi technique, address the length extrapolation challenge exclusively through the implementation of a single kernel function, which introduces a constant bias to every post-softmax attention scores according to their distance. These approaches do not investigate or employ multiple kernel functions to address the extrapolation challenge. Drawing on the ALiBi approach, this study proposes a novel relative positional encoding method, called MEP, which employs a weighted average to combine distinct kernel functions(such as the exponential kernel and the Gaussian kernel) to generate a bias that is applied to post-softmax attention scores. Initially, the framework utilizes various kernel functions to construct multiple kernel functions. Each kernel function adheres to a consistent mean weight coefficient, harnessing the synergistic advantages of different kernels to formulate an innovative bias function. Subsequently, specific slopes are tailored for each kernel function, applying penalties at varying rates, to enhance the model's extrapolation capabilities. Finally, this bias is seamlessly incorporated as a penalty to the post-softmax scores. We present two distinct versions of our method: a parameter-free variant that requires no new learnable parameters, which enhances length extrapolation capabilities without compromising training efficiency, and a parameterized variant capable of integrating state-of-the-art techniques. Empirical evaluations across diverse datasets have demonstrated that both variants of our method achieve state-of-the-art performance, outperforming traditional parameter-free and parameterized approaches.

KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation

MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation

Exploring Transformer Extrapolation

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation

On the token distance modeling ability of higher RoPE attention dimension

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective

CLEX: Continuous Length Extrapolation for Large Language Models

Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs

Scaling Laws of RoPE-based Extrapolation.

Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization

Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models.

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Linearized Relative Positional Encoding

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models

HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation

Round and Round We Go! What makes Rotary Positional Encodings useful?

The Impact of Positional Encoding on Length Generalization in Transformers