Abstract:Memory replay, which stores a subset of historical data from previous tasks to replay while learning new tasks, exhibits state-of-the-art performance for various continual learning applications on the Euclidean data. While topological information plays a critical role in characterizing graph data, existing memory replay-based graph learning techniques only store individual nodes for replay and do not consider their associated edge information. To this end, based on the message-passing mechanism in graph neural networks (GNNs), we present the Ricci curvature-based graph sparsification technique to perform continual graph representation learning. Specifically, we first develop the subgraph episodic memory (SEM) to store the topological information in the form of computation subgraphs. Next, we sparsify the subgraphs such that they only contain the most informative structures (nodes and edges). The informativeness is evaluated with the Ricci curvature, a theoretically justified metric to estimate the contribution of neighbors to represent a target node. In this way, we can reduce the memory consumption of a computation subgraph from O(dL) to O(1) and enable GNNs to fully utilize the most informative topological information for memory replay. Besides, to ensure the applicability on large graphs, we also provide the theoretically justified surrogate for the Ricci curvature in the sparsification process, which can greatly facilitate the computation. Finally, our empirical studies show that SEM outperforms state-of-the-art approaches significantly on four different public datasets. Unlike existing methods, which mainly focus on task incremental learning (task-IL) setting, SEM also succeeds in the challenging class incremental learning (class-IL) setting in which the model is required to distinguish all learned classes without task indicators and even achieves comparable performance to joint training, which is the performance upper bound for continual learning.

Graph Continual Learning with Debiased Lossless Memory Replay

Progressive Learning without Forgetting

Debiased Graph Contrastive Learning.

Sparsified Subgraph Memory for Continual Graph Representation Learning

On the Limitation and Experience Replay for GNNs in Continual Learning

Reinforced Continual Learning for Graphs

Online Continual Learning with Declarative Memory

Memory Enhanced Replay for Continual Learning

Overcoming Catastrophic Forgetting in Graph Neural Networks with Experience Replay

E-CGL: An Efficient Continual Graph Learner

Streaming Graph Neural Networks Via Continual Learning

Graph Memory Learning: Imitating Lifelong Remembering and Forgetting of Brain Networks

Federated Continual Graph Learning

Saliency-Guided Hidden Associative Replay for Continual Learning

Long Short-Term Graph Memory Against Class-imbalanced Over-smoothing

Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes

Ricci Curvature-Based Graph Sparsification for Continual Graph Representation Learning.

Sufficient Subgraph Embedding Memory for Continual Graph Representation Learning

Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning

Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning

Adaptive online continual multi-view learning