Abstract:Ensembles, which employ a set of classifiers to enhance classification accuracy collectively, are crucial in the era of big data. However, although there is general agreement that the relation between ensemble size and its prediction accuracy, the exact nature of this relationship is still unknown. We introduce a novel perspective, rooted in the linear independence of classifier's votes, to analyze the interplay between ensemble size and prediction accuracy. This framework reveals a theoretical link, consequently proposing an ensemble size based on this relationship. Our study builds upon a geometric framework and develops a series of theorems. These theorems clarify the role of linear dependency in crafting ensembles. We present a method to determine the minimum ensemble size required to ensure a target probability of linearly independent votes among component classifiers. Incorporating real and synthetic datasets, our empirical results demonstrate a trend: increasing the number of classifiers enhances accuracy, as predicted by our theoretical insights. However, we also identify a point of diminishing returns, beyond which additional classifiers provide diminishing improvements in accuracy. Surprisingly, the calculated ideal ensemble size deviates from empirical results for certain datasets, emphasizing the influence of other factors. This study opens avenues for deeper investigations into the complex dynamics governing ensemble design and offers guidance for constructing efficient and effective ensembles in practical scenarios.

Ensemble learning of linear perceptron; Online learning theory

Statistical Mechanics of Online Learning for Ensemble Teachers

Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule

On-line Learning of an Unlearnable True Teacher through Mobile Ensemble Teachers

On-line learning and generalisation in coupled perceptrons

A Local Online Learning Approach for Non-linear Data.

Optimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning

Leveraging Linear Independence of Component Classifiers: Optimizing Size and Prediction Accuracy for Online Ensembles

Statistical Mechanics of On-line Learning When a Moving Teacher Goes around an Unlearnable True Teacher

Generalization ability of a perceptron with non-monotonic transfer function

Learning curves for the multi-class teacher-student perceptron

Generalization Errors of the Simple Perceptron

Electronic structure study by means of X-ray spectroscopy and theoretical calculations of the "ferric star" single molecule magnet

On-line learning through simple perceptron with a margin

An Ensemble Model For Error Modeling With Pseudoinverse Learning Algorithm

Convergence of Unregularized Online Learning Algorithms

Statistical Learning Theory for Neural Operators

Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron

Fast and Strong Convergence of Online Learning Algorithms.

Analysis of On-Line Learning when a Moving Teacher Goes around a True Teacher

A Parallelizable Bayesian Ensemble Online Learning Algorithm