Abstract:Fully exploiting the learning capacity of neural networks requires overparameterized dense networks. On the other side, directly training sparse neural networks typically results in unsatisfactory performance. Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. Concretely, it claims there exist winning tickets from a randomly initialized network found by iterative magnitude pruning and preserving promising trainability (or we say being in trainable condition). In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark, then go from a complementary direction to articulate the Dual Lottery Ticket Hypothesis (DLTH): Randomly selected subnetworks from a randomly initialized dense network can be transformed into a trainable condition and achieve admirable performance compared with LTH -- random tickets in a given lottery pool can be transformed into winning tickets. Specifically, by using uniform-randomly selected subnetworks to represent the general cases, we propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH. Concretely, we introduce a regularization term to borrow learning capacity and realize information extrusion from the weights which will be masked. After finishing the transformation for the randomly selected subnetworks, we conduct the regular finetuning to evaluate the model using fair comparisons with LTH and other strong baselines. Extensive experiments on several public datasets and comparisons with competitive approaches validate our DLTH as well as the effectiveness of the proposed model RST. Our work is expected to pave a way for inspiring new research directions of sparse network training in the future. Our code is available at <a class="link-external link-https" href="https://github.com/yueb17/DLTH" rel="external noopener nofollow">this https URL</a>.

What's Hidden in a Randomly Weighted Neural Network?

Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks

Randomly Initialized Subnetworks with Iterative Weight Recycling

Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing

Efficient Design of Neural Networks with Random Weights

You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets

Residual Random Neural Networks

A Powerful Generative Model Using Random Weights for the Deep Image Representation

Multi-Activation Hidden Units for Neural Networks with Random Weights

Randomness in Neural Networks: an Overview

Random matrix analysis of deep neural network weight matrices

Randomness in Deconvolutional Networks for Visual Representation

Exploring Randomly Wired Neural Networks for Image Recognition

On Learnable Parameters of Optimal and Suboptimal Deep Learning Models

Randomnet: clustering time series using untrained deep neural networks

An Insect-Inspired Randomly, Weighted Neural Network with Random Fourier Features For Neuro-Symbolic Relational Learning

Revealing the Utilized Rank of Subspaces of Learning in Neural Networks

Random ReLU Neural Networks as Non-Gaussian Processes

Insights into Randomized Algorithms for Neural Networks: Practical Issues and Common Pitfalls.

Dual Lottery Ticket Hypothesis