Graph Neural Aggregation-diffusion with Metastability

Kaiyuan Cui,Xinyan Wang,Zicheng Zhang,Weichen Zhao

2024-03-29

Abstract:Continuous graph neural models based on differential equations have expanded the architecture of graph neural networks (GNNs). Due to the connection between graph diffusion and message passing, diffusion-based models have been widely studied. However, diffusion naturally drives the system towards an equilibrium state, leading to issues like over-smoothing. To this end, we propose GRADE inspired by graph aggregation-diffusion equations, which includes the delicate balance between nonlinear diffusion and aggregation induced by interaction potentials. The node representations obtained through aggregation-diffusion equations exhibit metastability, indicating that features can aggregate into multiple clusters. In addition, the dynamics within these clusters can persist for long time periods, offering the potential to alleviate over-smoothing effects. This nonlinear diffusion in our model generalizes existing diffusion-based models and establishes a connection with classical GNNs. We prove that GRADE achieves competitive performance across various benchmarks and alleviates the over-smoothing issue in GNNs evidenced by the enhanced Dirichlet energy.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The main focus of this paper is the problem of oversmoothing in Graph Neural Networks (GNNs). Oversmoothing refers to the phenomenon where the node representations in GNNs become overly consistent, leading to a loss in expressive power. The paper points out that continuous GNNs based on differential equations naturally drive the system towards an equilibrium state through the diffusion process, resulting in oversmoothing. To address this issue, the paper proposes a new model called GRADE, inspired by the aggregation-diffusion equation. GRADE incorporates nonlinear diffusion and aggregation induced by interaction potential, which allows the node representations to form multiple clusters and avoid oversmoothing. In this way, the model can maintain dynamics over long periods of time and have better expressive power. Experimental results demonstrate that GRADE performs competitively on various benchmark tasks and can alleviate the oversmoothing challenge in GNNs.

Graph Neural Aggregation-diffusion with Metastability

Flexible Graph Neural Diffusion with Latent Class Representation Learning

Model Degradation Hinders Deep Graph Neural Networks

Re-Think and Re-Design Graph Neural Networks in Spaces of Continuous Graph Diffusion Functionals

Adaptive Graph Diffusion Networks

Graph Neural Diffusion Networks for Semi-supervised Learning

Advective Diffusion Transformers for Topological Generalization in Graph Learning

Diffusion Improves Graph Learning

A Unified View on Graph Neural Networks as Graph Signal Denoising

Sparse Training of Discrete Diffusion Models for Graph Generation

Diffuse and Smooth: Beyond Truncated Receptive Field for Scalable and Adaptive Graph Representation Learning.

Graph Anisotropic Diffusion

Deep Graph Neural Networks via Flexible Subgraph Aggregation

From Continuous Dynamics to Graph Neural Networks: Neural Diffusion and Beyond

Message gain aggregation architecture: a scalable graph neural network for combining large-scale neighborhoods

Graph Anomaly Detection with Diffusion Model-Based Graph Enhancement (student Abstract)

Diffusion induced graph representation learning.

Graph Neural Reaction Diffusion Models

Autoregressive Diffusion Model for Graph Generation

Stability of Aggregation Graph Neural Networks

Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs