Abstract:Training deep neural networks is a very demanding task, especially challenging is how to adapt architectures to improve the performance of trained models. We can find that sometimes, shallow networks generalize better than deep networks, and the addition of more layers results in higher training and test errors. The deep residual learning framework addresses this degradation problem by adding skip connections to several neural network layers. It would at first seem counter-intuitive that such skip connections are needed to train deep networks successfully as the expressivity of a network would grow exponentially with depth. In this paper, we first analyze the flow of information through neural networks. We introduce and evaluate the batch-entropy which quantifies the flow of information through each layer of a neural network. We prove empirically and theoretically that a positive batch-entropy is required for gradient descent-based training approaches to optimize a given loss function successfully. Based on those insights, we introduce batch-entropy regularization to enable gradient descent-based training algorithms to optimize the flow of information through each hidden layer individually. With batch-entropy regularization, gradient descent optimizers can transform untrainable networks into trainable networks. We show empirically that we can therefore train a "vanilla" fully connected network and convolutional neural network -- no skip connections, batch normalization, dropout, or any other architectural tweak -- with 500 layers by simply adding the batch-entropy regularization term to the loss function. The effect of batch-entropy regularization is not only evaluated on vanilla neural networks, but also on residual networks, autoencoders, and also transformer models over a wide range of computer vision as well as natural language processing tasks.

Shake-Shake regularization

Shakedrop Regularization for Deep Residual Learning

Shakeout: A New Approach to Regularized Deep Neural Network Training

SparseConnect: Regularising CNNs on Fully Connected Layers

PatchShuffle Regularization.

Weight-Sharing Regularization

Stochastic Layer-Wise Shuffle: A Good Practice to Improve Vision Mamba Training

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Extremely Simple Activation Shaping for Out-of-Distribution Detection

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint.

Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations

Alleviating Representational Shift for Continual Fine-tuning.

Convolutional Neural Networks With Dynamic Regularization

Variance-Covariance Regularization Improves Representation Learning

REVE: Regularizing Deep Learning with Variational Entropy Bound

Stochastic Normalization.

Swapout: Learning an ensemble of deep architectures

Patch-level Neighborhood Interpolation: A General and Effective Graph-based Regularization Strategy

Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization

Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization

Class Regularization: Improve Few-shot Image Classification by Reducing Meta Shift