Abstract:Despite Graph Neural Networks demonstrating considerable promise in graph representation learning tasks, GNNs predominantly face significant issues with over-fitting and over-smoothing as they go deeper as models of computer vision realm. In this work, we conduct a systematic study of deeper GNN research trajectories. Our findings indicate that the current success of deep GNNs primarily stems from (I) the adoption of innovations from CNNs, such as residual/skip connections, or (II) the tailor-made aggregation algorithms like DropEdge. However, these algorithms often lack intrinsic interpretability and indiscriminately treat all nodes within a given layer in a similar manner, thereby failing to capture the nuanced differences among various nodes. To this end, we introduce the Snowflake Hypothesis -- a novel paradigm underpinning the concept of ``one node, one receptive field''. The hypothesis draws inspiration from the unique and individualistic patterns of each snowflake, proposing a corresponding uniqueness in the receptive fields of nodes in the GNNs. We employ the simplest gradient and node-level cosine distance as guiding principles to regulate the aggregation depth for each node, and conduct comprehensive experiments including: (1) different training schemes; (2) various shallow and deep GNN backbones, and (3) various numbers of layers (8, 16, 32, 64) on multiple benchmarks (six graphs including dense graphs with millions of nodes); (4) compare with different aggregation strategies. The observational results demonstrate that our hypothesis can serve as a universal operator for a range of tasks, and it displays tremendous potential on deep GNNs. It can be applied to various GNN frameworks, enhancing its effectiveness when operating in-depth, and guiding the selection of the optimal network depth in an explainable and generalizable way.

Universal Deep GNNs: Rethinking Residual Connection in GNNs from a Path Decomposition Perspective for Preventing the Over-smoothing

Deeper-GXX: Deepening Arbitrary GNNs

Beyond Over-smoothing: Uncovering the Trainability Challenges in Deep Graph Neural Networks

Beyond smoothness: A general optimization framework for graph neural networks with negative Laplacian regularization

Model Degradation Hinders Deep Graph Neural Networks

DRGCN: Dynamic Evolving Initial Residual for Deep Graph Convolutional Networks

DeGNN: Improving Graph Neural Networks with Graph Decomposition

Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs

The Snowflake Hypothesis: Training Deep GNN with One Node One Receptive field

DeGNN: Characterizing and Improving Graph Neural Networks with Graph Decomposition.

Deep Graph Neural Networks via Posteriori-Sampling-based Node-Adaptive Residual Module

DII-GCN: Dropedge Based Deep Graph Convolutional Networks

Graph Neural Networks Do Not Always Oversmooth

Decoupling the Depth and Scope of Graph Neural Networks

Deep Graph Neural Networks via Flexible Subgraph Aggregation

Graph Elimination Networks

AGNN: Alternating Graph-Regularized Neural Networks to Alleviate Over-Smoothing

A Unified View on Graph Neural Networks as Graph Signal Denoising

Simple and Deep Graph Convolutional Networks

Tackling Over-Smoothing for General Graph Convolutional Networks

On the Trade-off between Over-smoothing and Over-squashing in Deep Graph Neural Networks