Abstract:Training deep neural networks is a very demanding task, especially challenging is how to adapt architectures to improve the performance of trained models. We can find that sometimes, shallow networks generalize better than deep networks, and the addition of more layers results in higher training and test errors. The deep residual learning framework addresses this degradation problem by adding skip connections to several neural network layers. It would at first seem counter-intuitive that such skip connections are needed to train deep networks successfully as the expressivity of a network would grow exponentially with depth. In this paper, we first analyze the flow of information through neural networks. We introduce and evaluate the batch-entropy which quantifies the flow of information through each layer of a neural network. We prove empirically and theoretically that a positive batch-entropy is required for gradient descent-based training approaches to optimize a given loss function successfully. Based on those insights, we introduce batch-entropy regularization to enable gradient descent-based training algorithms to optimize the flow of information through each hidden layer individually. With batch-entropy regularization, gradient descent optimizers can transform untrainable networks into trainable networks. We show empirically that we can therefore train a "vanilla" fully connected network and convolutional neural network -- no skip connections, batch normalization, dropout, or any other architectural tweak -- with 500 layers by simply adding the batch-entropy regularization term to the loss function. The effect of batch-entropy regularization is not only evaluated on vanilla neural networks, but also on residual networks, autoencoders, and also transformer models over a wide range of computer vision as well as natural language processing tasks.

Regularizing Deep Networks Using Efficient Layerwise Adversarial Training

Improving Model Robustness Against Adversarial Examples with Redundant Fully Connected Layer.

Regularizers for Single-step Adversarial Training

LDN-RC: a Lightweight Denoising Network with Residual Connection to Improve Adversarial Robustness

Network as Regularization for Training Deep Neural Networks: Framework, Model and Performance

Regularization for Adversarial Robust Learning

Layer-wise Regularized Adversarial Training using Layers Sustainability Analysis (LSA) framework

Towards Robust Training of Neural Networks by Regularizing Adversarial Gradients

Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness

Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization

Layer-wise Adversarial Defense: an ODE Perspective

Regularization properties of adversarially-trained linear regression

RDAT: an efficient regularized decoupled adversarial training mechanism

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint.

A Brain-Inspired Regularizer for Adversarial Robustness

Training Robust Deep Neural Networks via Adversarial Noise Propagation

On Connections Between Regularizations for Improving DNN Robustness

Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients

Improved Adversarial Training Through Adaptive Instance-wise Loss Smoothing

Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization

Singular Regularization with Information Bottleneck Improves Model's Adversarial Robustness