Abstract:Dynamic graph embedding has emerged as a very effective technique for addressing diverse temporal graph analytic tasks (i.e., link prediction, node classification, recommender systems, anomaly detection, and graph generation) in various applications. Such temporal graphs exhibit heterogeneous transient dynamics, varying time intervals, and highly evolving node features throughout their evolution. Hence, incorporating long-range dependencies from the historical graph context plays a crucial role in accurately learning their temporal dynamics. In this paper, we develop a graph embedding model with uncertainty quantification, TransformerG2G, by exploiting the advanced transformer encoder to first learn intermediate node representations from its current state ($t$) and previous context (over timestamps [$t-1, t-l$], $l$ is the length of context). Moreover, we employ two projection layers to generate lower-dimensional multivariate Gaussian distributions as each node's latent embedding at timestamp $t$. We consider diverse benchmarks with varying levels of ``novelty" as measured by the TEA (Temporal Edge Appearance) plots. Our experiments demonstrate that the proposed TransformerG2G model outperforms conventional multi-step methods and our prior work (DynG2G) in terms of both link prediction accuracy and computational efficiency, especially for high degree of novelty. Furthermore, the learned time-dependent attention weights across multiple graph snapshots reveal the development of an automatic adaptive time stepping enabled by the transformer. Importantly, by examining the attention weights, we can uncover temporal dependencies, identify influential elements, and gain insights into the complex interactions within the graph structure. For example, we identified a strong correlation between attention weights and node degree at the various stages of the graph topology evolution.

Temporal Graph Transformer for Dynamic Network

Learning Continuous Dynamic Network Representation with Transformer-Based Temporal Graph Neural Network.

TPGNN: Learning High-order Information in Dynamic Graphs via Temporal Propagation

Temporal Graph Networks for Deep Learning on Dynamic Graphs

Spatial-temporal Graph Convolutional Networks with Diversified Transformation for Dynamic Graph Representation Learning

Temporal Aggregation and Propagation Graph Neural Networks for Dynamic Representation

Temporal Graph Network for continuous-time dynamic event sequence

Transition Propagation Graph Neural Networks for Temporal Networks

Graph Transformer Networks Based Text Representation.

ETC: Efficient Training of Temporal Graph Neural Networks over Large-Scale Dynamic Graphs

Towards Open Temporal Graph Neural Networks

TimeSGN: Scalable and Effective Temporal Graph Neural Network

Dynamic Graph Representation Learning via Edge Temporal States Modeling and Structure-reinforced Transformer

Continuous Temporal Graph Networks for Event-Based Graph Data

Retrofitting Temporal Graph Neural Networks with Transformer

K-Truss Based Temporal Graph Convolutional Network for Dynamic Graphs.

Dynamic Graph Transformer with Correlated Spatial-Temporal Positional Encoding

K-Core based Temporal Graph Convolutional Network for Dynamic Graphs

Scalable and Effective Temporal Graph Representation Learning With Hyperbolic Geometry

TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers

DTFormer: A Transformer-Based Method for Discrete-Time Dynamic Graph Representation Learning