Abstract:Dynamic graph embedding has emerged as a very effective technique for addressing diverse temporal graph analytic tasks (i.e., link prediction, node classification, recommender systems, anomaly detection, and graph generation) in various applications. Such temporal graphs exhibit heterogeneous transient dynamics, varying time intervals, and highly evolving node features throughout their evolution. Hence, incorporating long-range dependencies from the historical graph context plays a crucial role in accurately learning their temporal dynamics. In this paper, we develop a graph embedding model with uncertainty quantification, TransformerG2G, by exploiting the advanced transformer encoder to first learn intermediate node representations from its current state ($t$) and previous context (over timestamps [$t-1, t-l$], $l$ is the length of context). Moreover, we employ two projection layers to generate lower-dimensional multivariate Gaussian distributions as each node's latent embedding at timestamp $t$. We consider diverse benchmarks with varying levels of ``novelty" as measured by the TEA (Temporal Edge Appearance) plots. Our experiments demonstrate that the proposed TransformerG2G model outperforms conventional multi-step methods and our prior work (DynG2G) in terms of both link prediction accuracy and computational efficiency, especially for high degree of novelty. Furthermore, the learned time-dependent attention weights across multiple graph snapshots reveal the development of an automatic adaptive time stepping enabled by the transformer. Importantly, by examining the attention weights, we can uncover temporal dependencies, identify influential elements, and gain insights into the complex interactions within the graph structure. For example, we identified a strong correlation between attention weights and node degree at the various stages of the graph topology evolution.

Spacetime $E(n)$-Transformer: Equivariant Attention for Spatio-temporal Graphs

TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers

Continuous Spatiotemporal Transformers

Spectral Transform Forms Scalable Transformer

Concurrent Transformer for Spatial-Temporal Graph Modeling

Spiking Transformer with Spatial-Temporal Attention

Equivariant Spatio-Temporal Attentive Graph Networks to Simulate Physical Dynamics

Reach the Remote Neighbors: Dual-Encoding Transformer for Graphs

Structure-Preserving Transformers for Sequences of SPD Matrices

Adaptive Graph Spatial-Temporal Transformer Network for Traffic Flow Forecasting

Gauge Equivariant Transformer.

Transformer Embeddings of Irregularly Spaced Events and Their Participants

Space or time for video classification transformers

Equivariant Transformer is all you need

TE-Spikformer:Temporal-enhanced spiking neural network with transformer

Supra-Laplacian Encoding for Transformer on Dynamic Graphs

ETSformer: Exponential Smoothing Transformers for Time-series Forecasting

Graph Transformers without Positional Encodings

Expressing Multivariate Time Series as Graphs with Time Series Attention Transformer

DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability

Translation Equivariant Transformer Neural Processes