Abstract:Many positional encodings (PEs) are designed to exhibit long-term decay, based on an entrenched and long-standing inductive opinion: tokens farther away from the current position carry less relevant information. We argue that long-term decay is outdated in the era of LLMs, as LLMs are now applied to tasks demanding precise retrieval of in-context information from arbitrary positions. Firstly, we present empirical analyses on various PEs, demonstrating that models inherently learn attention with only a local-decay pattern while forming a U-shape pattern globally, contradicting the principle of long-term decay. Furthermore, we conduct a detailed analysis of rotary position encoding (RoPE, a prevalent relative positional encoding in LLMs), and found that the U-shape attention is caused by some learned components, which are also the key factor limiting RoPE's expressiveness and <a class="link-external link-http" href="http://extrapolation.Inspired" rel="external noopener nofollow">this http URL</a> by these insights, we propose High-frequency rotary Position Encoding (HoPE). HoPE replaces the specific components in RoPE with position-independent ones, retaining only high-frequency signals, which also breaks the principle of long-term decay in theory. HoPE achieves two major advantages: (1) Without constraints imposed by long-term decay, contradictory factors that limit spontaneous attention optimization and model extrapolation performance are removed. (2) Components representing positions and semantics are are optimized. These enhances model's context awareness and extrapolation, as validated by extensive experiments.

What Are Good Positional Encodings for Directed Graphs?

Position Encoding for Heterogeneous Graph Neural Networks

GeniePath: Graph Neural Networks with Adaptive Receptive Paths

On the Stability of Expressive Positional Encodings for Graphs

Bridging Graph Position Encodings for Transformers with Weighted Graph-Walking Automata

Graph Neural Networks with Learnable Structural and Positional Representations

Graph Positional and Structural Encoder

Discrete Embedding for Attributed Graphs

Comparing Graph Transformers via Positional Encodings

Direction Aware Positional and Structural Encoding for Directed Graph Neural Networks

GRPE: Relative Positional Encoding for Graph Transformer

Graph Transformers without Positional Encodings

Rewiring with Positional Encodings for Graph Neural Networks

Range-aware Positional Encoding via High-order Pretraining: Theory and Practice

HyPE-GT: where Graph Transformers meet Hyperbolic Positional Encodings

Transformers Meet Directed Graphs

PINE: Universal Deep Embedding for Graph Nodes via Partial Permutation Invariant Set Functions

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation

Structural and positional ensembled encoding for Graph Transformer

What functions can Graph Neural Networks compute on random graphs? The role of Positional Encoding

Graphrpe: Relative Position Encoding Graph Transformer for 3d Human Pose Estimation