Abstract:Graph neural networks (GNNs) are widely used for analyzing graph-structural data and solving graph-related tasks due to their powerful expressiveness. However, existing off-the-shelf GNN-based models usually consist of no more than three layers. Deeper GNNs usually suffer from severe performance degradation due to several issues including the infamous “over-smoothing” issue, which restricts the further development of GNNs. In this article, we investigate the over-smoothing issue in deep GNNs. We discover that over-smoothing not only results in indistinguishable embeddings of graph nodes, but also alters and even corrupts their semantic structures, dubbed semantic over-smoothing. Existing techniques, e.g., graph normalization, aim at handling the former concern, but neglect the importance of preserving the semantic structures in the spatial domain, which hinders the further improvement of model performance. To alleviate the concern, we propose a cluster-keeping sparse aggregation strategy to preserve the semantic structure of embeddings in deep GNNs (especially for spatial GNNs). Particularly, our strategy heuristically redistributes the extent of aggregations for all the nodes from layers, instead of aggregating them equally, so that it enables aggregate concise yet meaningful information for deep layers. Without any bells and whistles, it can be easily implemented as a plug-and-play structure of GNNs via weighted residual connections. Last, we analyze the over-smoothing issue on the GNNs with weighted residual structures and conduct experiments to demonstrate the performance comparable to the state-of-the-arts.

Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs

Beyond smoothness: A general optimization framework for graph neural networks with negative Laplacian regularization

Graph Neural Networks Do Not Always Oversmooth

Motif-induced Graph Normalization

Curriculum-Enhanced Residual Soft An-Isotropic Normalization for Over-smoothness in Deep GNNs

A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks

Taming Gradient Oversmoothing and Expansion in Graph Neural Networks

Another Perspective of Over-Smoothing: Alleviating Semantic Over-Smoothing in Deep GNNs

A Survey on Oversmoothing in Graph Neural Networks

Resisting Over-Smoothing in Graph Neural Networks Via Dual-Dimensional Decoupling

Graph Neural Networks Need Cluster-Normalize-Activate Modules

Avoiding Oversmoothing in Deep Graph Neural Networks: A Multiplicative Ergodic Analysis

ATNPA: A Unified View of Oversmoothing Alleviation in Graph Neural Networks

Oversquashing in GNNs through the lens of information contraction and graph expansion

Gentle Normalization and Translation in Graph Neural Network for Few-shot Learning

Improving Expressivity of GNNs with Subgraph-specific Factor Embedded Normalization

GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

GRANOLA: Adaptive Normalization for Graph Neural Networks

Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View

Tackling Long-tailed Distribution Issue in Graph Neural Networks via Normalization

Learning graph normalization for graph neural networks