Abstract:Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. Mainstream solutions to this problem rely mainly on knowledge distillation, which involves a two-stage procedure: first training a large teacher model and then distilling it to improve the generalization ability of smaller ones. In this work, we introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers, namely, slimmable networks for contrastive self-supervised learning (SlimCLR). A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks, including small ones with low computation costs. However, interference between weight-sharing networks leads to severe performance degradation in self-supervised cases, as evidenced by gradient magnitude imbalance and gradient direction divergence. The former indicates that a small proportion of parameters produce dominant gradients during backpropagation, while the main parameters may not be fully optimized. The latter shows that the gradient direction is disordered, and the optimization process is unstable. To address these issues, we introduce three techniques to make the main parameters produce dominant gradients and sub-networks have consistent outputs. These techniques include slow start training of sub-networks, online distillation, and loss re-weighting according to model sizes. Furthermore, theoretical results are presented to demonstrate that a single slimmable linear layer is sub-optimal during linear evaluation. Thus a switchable linear probe layer is applied during linear evaluation. We instantiate SlimCLR with typical contrastive learning frameworks and achieve better performance than previous arts with fewer parameters and FLOPs. The code is at <a class="link-external link-https" href="https://github.com/mzhaoshuai/SlimCLR" rel="external noopener nofollow">this https URL</a>.

Performance of Three Slim Variants of The Long Short-Term Memory (LSTM) Layer

SLIM LSTMs

Slim LSTM networks: LSTM_6 and LSTM_C6

Simplified Long Short-term Memory Recurrent Neural Networks: part III

A Performance Review of Recurrent Neural Networks Long Short-Term Memory (LSTM)

LSTM: A Search Space Odyssey

Simplified Gating in Long Short-term Memory (LSTM) Recurrent Neural Networks

Natural Language Inference Using Lstm Model With Sentence Fusion

Deep Multi-Kernel Convolutional LSTM Networks and an Attention-Based Mechanism for Videos

Slimmable Networks for Contrastive Self-supervised Learning

Towards Efficient Recurrent Architectures: A Deep LSTM Neural Network Applied to Speech Enhancement and Recognition

Approximate Computing for Long Short Term Memory (LSTM) Neural Networks

LSTM Pose Machines.

Towards Energy-Efficient, Low-Latency and Accurate Spiking LSTMs

SlimML: Removing Non-Critical Input Data in Large-Scale Iterative Machine Learning

Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM

LLM Performance Predictors are good initializers for Architecture Search

Learning Efficient Convolutional Networks Through Network Slimming.

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs