Abstract:Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. Mainstream solutions to this problem rely mainly on knowledge distillation, which involves a two-stage procedure: first training a large teacher model and then distilling it to improve the generalization ability of smaller ones. In this work, we introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers, namely, slimmable networks for contrastive self-supervised learning (SlimCLR). A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks, including small ones with low computation costs. However, interference between weight-sharing networks leads to severe performance degradation in self-supervised cases, as evidenced by gradient magnitude imbalance and gradient direction divergence. The former indicates that a small proportion of parameters produce dominant gradients during backpropagation, while the main parameters may not be fully optimized. The latter shows that the gradient direction is disordered, and the optimization process is unstable. To address these issues, we introduce three techniques to make the main parameters produce dominant gradients and sub-networks have consistent outputs. These techniques include slow start training of sub-networks, online distillation, and loss re-weighting according to model sizes. Furthermore, theoretical results are presented to demonstrate that a single slimmable linear layer is sub-optimal during linear evaluation. Thus a switchable linear probe layer is applied during linear evaluation. We instantiate SlimCLR with typical contrastive learning frameworks and achieve better performance than previous arts with fewer parameters and FLOPs. The code is at <a class="link-external link-https" href="https://github.com/mzhaoshuai/SlimCLR" rel="external noopener nofollow">this https URL</a>.

Self-Delimiting Neural Networks

Efficient Structure Slimming for Spiking Neural Networks

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

IM-LIF: Improved Neuronal Dynamics with Attention Mechanism for Direct Training Deep Spiking Neural Network

Continual Learning with Deep Artificial Neurons

Learning Delays Through Gradients and Structure: Emergence of Spatiotemporal Patterns in Spiking Neural Networks

SLIM LSTMs

On neurodynamics with limiter function and linsker's developmental model

CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networks

Selfless Sequential Learning

slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks

Rethinking Deep Learning: Non-backpropagation and Non-optimization Machine Learning Approach Using Hebbian Neural Networks

Rethinking Deep Learning: Propagating Information in Neural Networks without Backpropagation and Statistical Optimization

Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via $\ell_1, \ell_0$, and transformed-$\ell_1$ Penalties

Pursing the Sparse Limitation of Spiking Deep Learning Structures

Neuron-centric Hebbian Learning

BrainTransformers: SNN-LLM

Slimmable Networks for Contrastive Self-supervised Learning

KLIF: An Optimized Spiking Neuron Unit for Tuning Surrogate Gradient Function

Bio-inspired, task-free continual learning through activity regularization