Abstract:Regularization is a very effective algorithm to solve overfitting problem in neural network, which improves the generalization ability of the model. However, their working mechanisms and the impact on the model performance have not been fully explored. In this paper, we study and analyze them using information bottleneck theory and one theory from human brain sensory system. We propose a metric to characterise the encoding length of hidden layers, named as AEntry value. Then, we implement extensive experiments on MNIST and FashionMNIST datasets with several commonly used regularization algorithms, and calculate the corresponding AEntry values. We analyze these results and obtain three conclusions. (1) The introduction of regularization influences the encoding of relative features with prediction task in neural network. The early stopping technique avoids introducing unrelated information with the task into the model by stopping the training process as an appropriate iterations. Laplace, Gaussian and Sparse Response regularizations compress the related representation and improve the performance of neural network by introducing the prior information into the model. In contrast, Dropout, Batch Normalization, and Layer Normalization increase the encoding length of features by adopting redundant representation to improve the performance. (2) The encoding of neural network does not satisfy the data processing inequality of information theory, which is mainly caused by redundant coding of extracted features. (3) The overfitting is caused by introducing irrelative information with the target. These results can give us insight into building more efficient regularization algorithm to improve the performance of neural network model.

A Comparative Study on Regularization Strategies for Embedding-based Neural Networks.

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

Regularized Structured Perceptron: A Case Study on Chinese Word Segmentation, POS Tagging and Parsing.

Improve Network Embeddings with Regularization.

Information Guided Regularization for Fine-tuning Language Models

Penetrating the influence of regularizations on neural network based on information bottleneck theory

Regularizing Deep Neural Networks by Ensemble-based Low-Level Sample-Variances Method

Network as Regularization for Training Deep Neural Networks: Framework, Model and Performance

Towards Better Understanding with Uniformity and Explicit Regularization of Embeddings in Embedding-based Neural Topic Models

LDA-Reg: Knowledge Driven Regularization using External Corpora

Recurrent Neural Network Regularization

Consistency of Neural Networks with Regularization

Learning Regularized Noise Contrastive Estimation for Robust Network Embedding.

Gram Regularization for Sparse and Disentangled Representation

Embedding with Autoencoder Regularization.

Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network

Learning Structured and Non-Redundant Representations with Deep Neural Networks

Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation

A Hybrid Regularization Approach for Random Vector Functional-Link Networks.

A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings

An Empirical Analysis of Different Sparse Penalties for Autoencoder in Unsupervised Feature Learning