Abstract:Graph Neural Networks (GNNs) have emerged as the leading deep learning models for graph-based representation learning. However, the training and inference of GNNs on large graphs remain resource-intensive, impeding their utility in real-world scenarios and curtailing their applicability in deeper and more sophisticated GNN architectures. To address this issue, the Graph Lottery Ticket (GLT) hypothesis assumes that GNN with random initialization harbors a pair of core subgraph and sparse subnetwork, which can yield comparable performance and higher efficiency to that of the original dense network and complete graph. Despite that GLT offers a new paradigm for GNN training and inference, existing GLT algorithms heavily rely on trial-and-error pruning rate tuning and scheduling, and adhere to an irreversible pruning paradigm that lacks elasticity. Worse still, current methods suffer scalability issues when applied to deep GNNs, as they maintain the same topology structure across all layers. These challenges hinder the integration of GLT into deeper and larger-scale GNN contexts. To bridge this critical gap, this paper introduces an $\textbf{A}$daptive, $\textbf{D}$ynamic, and $\textbf{A}$utomated framework for identifying $\textbf{G}$raph $\textbf{L}$ottery $\textbf{T}$ickets ($\textbf{AdaGLT}$). Our proposed method derives its key advantages and addresses the above limitations through the following three aspects: 1) tailoring layer-adaptive sparse structures for various datasets and GNNs, thus endowing it with the capability to facilitate deeper GNNs; 2) integrating the pruning and training processes, thereby achieving a dynamic workflow encompassing both pruning and restoration; 3) automatically capturing graph lottery tickets across diverse sparsity levels, obviating the necessity for extensive pruning parameter tuning. More importantly, we rigorously provide theoretical proofs to guarantee $\textbf{AdaGLT}$ to mitigate over-smoothing issues and obtain improved sparse structures in deep GNN scenarios. Extensive experiments demonstrate that $\textbf{AdaGLT}$ outperforms state-of-the-art competitors across multiple graph datasets of various scales and types, particularly in scenarios involving deep GNNs.

Rare Gems: Finding Lottery Tickets at Initialization

Winning Lottery Tickets in Deep Generative Models

The Elastic Lottery Ticket Hypothesis

A UNIFIED VIEW OF FINDING AND TRANSFORMING WINNING LOTTERY TICKETS

One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning

On the Existence of Universal Lottery Tickets

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets

Rethinking Graph Lottery Tickets: Graph Sparsity Matters

Searching Lottery Tickets in Graph Neural Networks: A Dual Perspective

Can We Find Strong Lottery Tickets in Generative Models?

Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning

Plant 'n' Seek: Can You Find the Winning Ticket?

Lottery Jackpots Exist in Pre-trained Models.

Dual Lottery Ticket Hypothesis

Graph Lottery Ticket Automated

When Layers Play the Lottery, all Tickets Win at Initialization

Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP

Randomly Initialized Subnetworks with Iterative Weight Recycling

Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis