Abstract:Fully exploiting the learning capacity of neural networks requires overparameterized dense networks. On the other side, directly training sparse neural networks typically results in unsatisfactory performance. Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. Concretely, it claims there exist winning tickets from a randomly initialized network found by iterative magnitude pruning and preserving promising trainability (or we say being in trainable condition). In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark, then go from a complementary direction to articulate the Dual Lottery Ticket Hypothesis (DLTH): Randomly selected subnetworks from a randomly initialized dense network can be transformed into a trainable condition and achieve admirable performance compared with LTH -- random tickets in a given lottery pool can be transformed into winning tickets. Specifically, by using uniform-randomly selected subnetworks to represent the general cases, we propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH. Concretely, we introduce a regularization term to borrow learning capacity and realize information extrusion from the weights which will be masked. After finishing the transformation for the randomly selected subnetworks, we conduct the regular finetuning to evaluate the model using fair comparisons with LTH and other strong baselines. Extensive experiments on several public datasets and comparisons with competitive approaches validate our DLTH as well as the effectiveness of the proposed model RST. Our work is expected to pave a way for inspiring new research directions of sparse network training in the future. Our code is available at <a class="link-external link-https" href="https://github.com/yueb17/DLTH" rel="external noopener nofollow">this https URL</a>.

Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

Data-Efficient Double-Win Lottery Tickets from Robust Pre-training

Finding the Dominant Winning Ticket in Pre-Trained Language Models

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training

The Diversified Ensemble Neural Network.

Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning

Developing parsimonious ensembles using ensemble diversity within a reinforcement learning framework

Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling

Prune and Tune Ensembles: Low-Cost Ensemble Learning with Sparse Independent Subnetworks

Winning Lottery Tickets in Deep Generative Models

Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP

One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

Efficient Diversity-Driven Ensemble for Deep Neural Networks

Dual Lottery Ticket Hypothesis

Playing Lottery Tickets with Vision and Language

LOFT: Finding Lottery Tickets through Filter-wise Training

Universality of Winning Tickets: A Renormalization Group Perspective

Learning Diverse Models for End-to-End Ensemble Tracking.

Sequential Bayesian Neural Subnetwork Ensembles

A Survey of Lottery Ticket Hypothesis

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning