Abstract:Discrete-Time Dynamic Graphs (DTDGs), which are prevalent in real-world implementations and notable for their ease of data acquisition, have garnered considerable attention from both academic researchers and industry practitioners. The representation learning of DTDGs has been extensively applied to model the dynamics of temporally changing entities and their evolving connections. Currently, DTDG representation learning predominantly relies on GNN+RNN architectures, which manifest the inherent limitations of both Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs). GNNs suffer from the over-smoothing issue as the models architecture goes deeper, while RNNs struggle to capture long-term dependencies effectively. GNN+RNN architectures also grapple with scaling to large graph sizes and long sequences. Additionally, these methods often compute node representations separately and focus solely on individual node characteristics, thereby overlooking the behavior intersections between the two nodes whose link is being predicted, such as instances where the two nodes appear together in the same context or share common neighbors. This paper introduces a novel representation learning method DTFormer for DTDGs, pivoting from the traditional GNN+RNN framework to a Transformer-based architecture. Our approach exploits the attention mechanism to concurrently process topological information within the graph at each timestamp and temporal dynamics of graphs along the timestamps, circumventing the aforementioned fundamental weakness of both GNNs and RNNs. Moreover, we enhance the model's expressive capability by incorporating the intersection relationships among nodes and integrating a multi-patching module. Extensive experiments conducted on six public dynamic graph benchmark datasets confirm our model's efficacy, achieving the SOTA performance.

GraKerformer: A Transformer With Graph Kernel for Unsupervised Graph Representation Learning

GTA: Graph Transformer Adapter

Do Transformers Really Perform Badly for Graph Representation?

KDLGT: A Linear Graph Transformer Framework Via Kernel Decomposition Approach.

Do Transformers Really Perform Bad for Graph Representation?

MTKGCformer: A Multi-train Transformer-based Representation Learning for Knowledge Graph Completion Task

HINormer: Representation Learning on Heterogeneous Information Networks with Graph Transformer.

TGformer: A Graph Transformer Framework for Knowledge Graph Embedding

KerGNNs: Interpretable Graph Neural Networks with Graph Kernels

DT4KGR: Decision transformer for fast and effective multi-hop reasoning over knowledge graphs

Graph-Aware Transformer: Is Attention All Graphs Need?

GraRep++: Flexible Learning Graph Representations with Weighted Global Structural Information.

DTFormer: A Transformer-Based Method for Discrete-Time Dynamic Graph Representation Learning

SpikeGraphormer: A High-Performance Graph Transformer with Spiking Graph Attention

VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections

Adaptive Multi-Neighborhood Attention based Transformer for Graph Representation Learning

Gradformer: Graph Transformer with Exponential Decay

Gransformer: Transformer-based Graph Generation

Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers

Transformer for Graphs: An Overview from Architecture Perspective

GTC: GNN-Transformer co-contrastive learning for self-supervised heterogeneous graph representation