Abstract:Fully exploiting the learning capacity of neural networks requires overparameterized dense networks. On the other side, directly training sparse neural networks typically results in unsatisfactory performance. Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. Concretely, it claims there exist winning tickets from a randomly initialized network found by iterative magnitude pruning and preserving promising trainability (or we say being in trainable condition). In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark, then go from a complementary direction to articulate the Dual Lottery Ticket Hypothesis (DLTH): Randomly selected subnetworks from a randomly initialized dense network can be transformed into a trainable condition and achieve admirable performance compared with LTH -- random tickets in a given lottery pool can be transformed into winning tickets. Specifically, by using uniform-randomly selected subnetworks to represent the general cases, we propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH. Concretely, we introduce a regularization term to borrow learning capacity and realize information extrusion from the weights which will be masked. After finishing the transformation for the randomly selected subnetworks, we conduct the regular finetuning to evaluate the model using fair comparisons with LTH and other strong baselines. Extensive experiments on several public datasets and comparisons with competitive approaches validate our DLTH as well as the effectiveness of the proposed model RST. Our work is expected to pave a way for inspiring new research directions of sparse network training in the future. Our code is available at <a class="link-external link-https" href="https://github.com/yueb17/DLTH" rel="external noopener nofollow">this https URL</a>.

Lottery Ticket Hypothesis for Attention Mechanism in Residual Convolutional Neural Network*

Efficient Attention Network: Accelerate Attention by Searching Where to Plug

A Survey of Lottery Ticket Hypothesis

Understanding Self-attention Mechanism via Dynamical System Perspective

The Elastic Lottery Ticket Hypothesis

Dual Lottery Ticket Hypothesis

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

A Generic Shared Attention Mechanism for Various Backbone Neural Networks

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

Successfully Applying Lottery Ticket Hypothesis to Diffusion Model

HAM: Hybrid Attention Module in Deep Convolutional Neural Networks for Image Classification

Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking Neural Network

Data Level Lottery Ticket Hypothesis for Vision Transformers

Loss-Based Attention for Interpreting Image-Level Prediction of Convolutional Neural Networks

Playing Lottery Tickets with Vision and Language

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training

A Simple and Light-Weight Attention Module for Convolutional Neural Networks

Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets

Switchable Self-attention Module