Abstract:Dynamic graph embedding has emerged as a very effective technique for addressing diverse temporal graph analytic tasks (i.e., link prediction, node classification, recommender systems, anomaly detection, and graph generation) in various applications. Such temporal graphs exhibit heterogeneous transient dynamics, varying time intervals, and highly evolving node features throughout their evolution. Hence, incorporating long-range dependencies from the historical graph context plays a crucial role in accurately learning their temporal dynamics. In this paper, we develop a graph embedding model with uncertainty quantification, TransformerG2G, by exploiting the advanced transformer encoder to first learn intermediate node representations from its current state ($t$) and previous context (over timestamps [$t-1, t-l$], $l$ is the length of context). Moreover, we employ two projection layers to generate lower-dimensional multivariate Gaussian distributions as each node's latent embedding at timestamp $t$. We consider diverse benchmarks with varying levels of ``novelty" as measured by the TEA (Temporal Edge Appearance) plots. Our experiments demonstrate that the proposed TransformerG2G model outperforms conventional multi-step methods and our prior work (DynG2G) in terms of both link prediction accuracy and computational efficiency, especially for high degree of novelty. Furthermore, the learned time-dependent attention weights across multiple graph snapshots reveal the development of an automatic adaptive time stepping enabled by the transformer. Importantly, by examining the attention weights, we can uncover temporal dependencies, identify influential elements, and gain insights into the complex interactions within the graph structure. For example, we identified a strong correlation between attention weights and node degree at the various stages of the graph topology evolution.

PGTNet: A Process Graph Transformer Network for Remaining Time Prediction of Business Process Instances

Business process remaining time prediction using explainable reachability graph from gated RNNs

Attention Please: What Transformer Models Really Learn for Process Prediction

Path Enhanced Bidirectional Graph Attention Network for Quality Prediction in Multistage Manufacturing Process

LSTM Networks for Data-Aware Remaining Time Prediction of Business Process Instances

A Novel Business Process Prediction Model Using a Deep Learning Method

PELP: Pioneer Event Log Prediction Using Sequence-to-Sequence Neural Networks

Graph Transformation Policy Network for Chemical Reaction Prediction

Modeling Multi-Step Scientific Processes with Graph Transformer Networks

TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers

GTHP: a novel graph transformer Hawkes process for spatiotemporal event prediction

Optimizing Graph Transformer Networks with Graph-based Techniques

Temporal Graph Networks for Deep Learning on Dynamic Graphs

Exploring the effect of context information on deep learning business process predictions

Process Drift Detection in Event Logs with Graph Convolutional Networks.

Decay Replay Mining to Predict Next Process Events

Dynamic spatial aware graph transformer for spatiotemporal traffic flow forecasting

Generalizing Graph Transformers Across Diverse Graphs and Tasks via Pre-Training on Industrial-Scale Data

Improving Predictive Process Monitoring Through Reachability Graph-Based Masking of Neural Networks

Learning Neural Point Processes with Latent Graphs

Graph Neural Point Process for Temporal Interaction Prediction.