Abstract:In this paper, the method of gaps, a technique for deriving closed-form expressions in terms of information measures for the generalization error of machine learning algorithms is introduced. The method relies on two central observations: $(a)$~The generalization error is an average of the variation of the expected empirical risk with respect to changes on the probability measure (used for expectation); and~$(b)$~these variations, also referred to as gaps, exhibit closed-form expressions in terms of information measures. The expectation of the empirical risk can be either with respect to a measure on the models (with a fixed dataset) or with respect to a measure on the datasets (with a fixed model), which results in two variants of the method of gaps. The first variant, which focuses on the gaps of the expected empirical risk with respect to a measure on the models, appears to be the most general, as no assumptions are made on the distribution of the datasets. The second variant develops under the assumption that datasets are made of independent and identically distributed data points. All existing exact expressions for the generalization error of machine learning algorithms can be obtained with the proposed method. Also, this method allows obtaining numerous new exact expressions, which improves the understanding of the generalization error; establish connections with other areas in statistics, e.g., hypothesis testing; and potentially, might guide algorithm designs.

Generalization Errors of the Simple Perceptron

Estimating Exact Form of Generalisation Errors

A Discussion of the Learning in a Simple Perceptron

Generalization ability of a perceptron with non-monotonic transfer function

Learning in A Higher-Order Simple Perceptron

The Generalization Error of Machine Learning Algorithms

Generalization Error of Generalized Linear Models in High Dimensions

Complexity of Error Hypersurfaces in Multilayer Perceptrons.

Ensemble learning of linear perceptron; Online learning theory

Learning curves for the multi-class teacher-student perceptron

On-line learning through simple perceptron with a margin

Perceptron Mistake Bounds

Complexity of error hypersurfaces in multilayer perceptrons with general multi-input and multi-output architecture

The Localized Generalization Error Model for Single Layer Perceptron Neural Network and Sigmoid Support Vector Machine.

Reserch on Perceptron Learning Algorithm

Application of the Error Function in Analyzing the Learning Dynamics Near Singularities of the Multilayer Perceptrons

A new approach to generalisation error of machine learning algorithms: Estimates and convergence

Generalised Perceptron Learning

A canonical ensemble approach to graded-response perceptrons

Generalization error for decision problems

The generalization error of the symmetric and scaled support vector machines.