Abstract:Network pruning is an effective approach to reduce network complexity with acceptable performance compromise. Existing studies achieve the sparsity of neural networks via time-consuming weight training or complex searching on networks with expanded width, which greatly limits the applications of network pruning. In this paper, we show that high-performing and sparse sub-networks without the involvement of weight training, termed "lottery jackpots", exist in pre-trained models with unexpanded width. Our presented lottery jackpots are traceable through empirical and theoretical outcomes. For example, we obtain a lottery jackpot that has only 10% parameters and still reaches the performance of the original dense VGGNet-19 without any modifications on the pre-trained weights on CIFAR-10. Furthermore, we improve the efficiency for searching lottery jackpots from two perspectives. First, we observe that the sparse masks derived from many existing pruning criteria have a high overlap with the searched mask of our lottery jackpot, among which, the magnitude-based pruning results in the most similar mask with ours. In compliance with this insight, we initialize our sparse mask using the magnitude-based pruning, resulting in at least 3× cost reduction on the lottery jackpot searching while achieving comparable or even better performance. Second, we conduct an in-depth analysis of the searching process for lottery jackpots. Our theoretical result suggests that the decrease in training loss during weight searching can be disturbed by the dependency between weights in modern networks. To mitigate this, we propose a novel short restriction method to restrict change of masks that may have potential negative impacts on the training loss, which leads to a faster convergence and reduced oscillation for searching lottery jackpots. Consequently, our searched lottery jackpot removes 90% weights in ResNet-50, while it easily obtains more than 70% top-1 accuracy using only 5 searching epochs on ImageNet.

When Layers Play the Lottery, all Tickets Win at Initialization

Rethinking the Value of Network Pruning

No Free Prune: Information-Theoretic Barriers to Pruning at Initialization

The Elastic Lottery Ticket Hypothesis

Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning

Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks

The Unreasonable Ineffectiveness of the Deeper Layers

Distilled Pruning: Using Synthetic Data to Win the Lottery

Lottery Ticket Preserves Weight Correlation: is It Desirable or Not?

Pruning neural networks without any data by iteratively conserving synaptic flow

Plant 'n' Seek: Can You Find the Winning Ticket?

A UNIFIED VIEW OF FINDING AND TRANSFORMING WINNING LOTTERY TICKETS

Efficient Weight Pruning using Pre-trained Lottery Jackpots

Rare Gems: Finding Lottery Tickets at Initialization

Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning

Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets

To Filter Prune, or to Layer Prune, That Is the Question

Exploring the Performance of Pruning Methods in Neural Networks: An Empirical Study of the Lottery Ticket Hypothesis

Concurrent Training and Layer Pruning of Deep Neural Networks

Lottery Jackpots Exist in Pre-trained Models.

Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis