Abstract:Deep neural networks have become the method of choice for solving many classification tasks, largely because they can fit very complex functions defined over raw data. The downside of such powerful learners is the danger of overfit. In this paper, we introduce a novel ensemble classifier for deep networks that effectively overcomes overfitting by combining models generated at specific intermediate epochs during training. Our method allows for the incorporation of useful knowledge obtained by the models during the overfitting phase without deterioration of the general performance, which is usually missed when early stopping is used. To motivate this approach, we begin with the theoretical analysis of a regression model, whose prediction -- that the variance among classifiers increases when overfit occurs -- is demonstrated empirically in deep networks in common use. Guided by these results, we construct a new ensemble-based prediction method, where the prediction is determined by the class that attains the most consensual prediction throughout the training epochs. Using multiple image and text classification datasets, we show that when regular ensembles suffer from overfit, our method eliminates the harmful reduction in generalization due to overfit, and often even surpasses the performance obtained by early stopping. Our method is easy to implement and can be integrated with any training scheme and architecture, without additional prior knowledge beyond the training set. It is thus a practical and useful tool to overcome overfit. Code is available at <a class="link-external link-https" href="https://github.com/uristern123/United-We-Stand-Using-Epoch-wise-Agreement-of-Ensembles-to-Combat-Overfit" rel="external noopener nofollow">this https URL</a>.

A Neural Network Model for the Overfitting Problem in Deep Learning

A Learning Algorithm for Solving the Over-Fitting Problem in BPANN

A Neural Network Model for General Minimax Problem

Research on Deep Neural Network Model Construction and Overfitting

Optimizing Convolutional Neural Network Performance by Mitigating Underfitting and Overfitting.

VECA: A Method for Detecting Overfitting in Neural Networks (Student Abstract)

An Overview of Overfitting and its Solutions

A systematic review on overfitting control in shallow and deep neural networks

United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfit

Towards an Understanding of Benign Overfitting in Neural Networks

Learning Structured and Non-Redundant Representations with Deep Neural Networks

A Neural Network Boosting Regression Model Based on XGBoost.

An Expanded Training Set Based Validation Method to Avoid Overfitting for Neural Network Classifier

The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks

Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting

Relearning Forgotten Knowledge: on Forgetting, Overfit and Training-Free Ensembles of DNNs

A Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

Network as Regularization for Training Deep Neural Networks: Framework, Model and Performance

On the Generalization Properties of Minimum-norm Solutions for Over-parameterized Neural Network Models