Optimal Linear Combination of Classifiers

Georgi Nalbantov,Svetoslav Ivanov

DOI: https://doi.org/10.48550/arXiv.2103.01109

IF: 5.414

2021-03-01

Machine Learning

Abstract:The question of whether to use one classifier or a combination of classifiers is a central topic in Machine Learning. We propose here a method for finding an optimal linear combination of classifiers derived from a bias-variance framework for the classification task.

What problem does this paper attempt to address?

Optimal Linear Combination Using Decision Reliability Of Individual Classifiers

Z Lu,Xq Ding

DOI: https://doi.org/10.1109/ISIMP.2001.925326

2001-01-01

Abstract:This paper presents a new combination method to adaptively assign combination weights according to the decision reliability, of each individual classifier; which is estimated from the posterior probabilities of the current sample. The model and training algorithm proposed in this paper are based on the linear combination model and mean squared errors criterion, but can also be extended to other combination methods and criterions. Experimental results on artificial data set and practical data set show that the proposed approach can achieve the best classification performance among other linear combination methods.
Optimal Classifier Fusion in a Non-Bayesian Probabilistic Framework

O.R. Terrades,E. Valveny,S. Tabbone

DOI: https://doi.org/10.1109/tpami.2008.224

IF: 23.6

2009-09-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:The combination of the output of classifiers has been one of the strategies used to improve classification rates in general purpose classification systems. Some of the most common approaches can be explained using the Bayes' formula. In this paper, we tackle the problem of the combination of classifiers using a non-Bayesian probabilistic framework. This approach permits us to derive two linear combination rules that minimize misclassification rates under some constraints on the distribution of classifiers. In order to show the validity of this approach we have compared it with other popular combination rules from a theoretical viewpoint using a synthetic data set, and experimentally using two standard databases: the MNIST handwritten digit database and the GREC symbol database. Results on the synthetic data set show the validity of the theoretical approach. Indeed, results on real data show that the proposed methods outperform other common combination schemes.

computer science, artificial intelligence,engineering, electrical & electronic
Combination of linear classifiers using score function -- analysis of possible combination strategies

Pawel Trajdos,Robert Burduk

DOI: https://doi.org/10.48550/arXiv.1905.09522

IF: 5.414

2019-05-23

Machine Learning

Abstract:In this work, we addressed the issue of combining linear classifiers using their score functions. The value of the scoring function depends on the distance from the decision boundary. Two score functions have been tested and four different combination strategies were investigated. During the experimental study, the proposed approach was applied to the heterogeneous ensemble and it was compared to two reference methods -- majority voting and model averaging respectively. The comparison was made in terms of seven different quality criteria. The result shows that combination strategies based on simple average, and trimmed average are the best combination strategies of the geometrical combination.
Decision combination in multiple classifier systems

Tin Kam Ho,J.J. Hull,S.N. Srihari

DOI: https://doi.org/10.1109/34.273716

IF: 23.6

1994-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:A multiple classifier system is a powerful solution to difficult pattern recognition problems involving large class sets and noisy input because it allows simultaneous use of arbitrary feature descriptors and classification procedures. Decisions by the classifiers can be represented as rankings of classifiers and different instances of a problem. The rankings can be combined by methods that either reduce or rerank a given set of classes. An intersection method and union method are proposed for class set reduction. Three methods based on the highest rank, the Borda count, and logistic regression are proposed for class set reranking. These methods have been tested in applications of degraded machine-printed characters and works from large lexicons, resulting in substantial improvement in overall correctness.<>

computer science, artificial intelligence,engineering, electrical & electronic
Methods of Linear Combination Based on Different Features

L Wang,K Chen,HS Chi

1997-01-01

Abstract:Linear Opinion Pools is a popular class of method for Combining Multiple Classifiers, which includes two cases: weights as veridical assumption and minimum error weights. This paper discusses these two cases based on different features, and applies them to the Text-Independent Speaker Identification. For the first case based on different features, a modified model of generalized finite mixture distribution is presented, which can adjust the coefficients according to the prior knowledge, while for the second case, a generalized ensemble method can be adopted directly. The experimental results show the effectiveness of the combination methods,using Linear Opinion Pools based on different features.
Linear Regression Based Combination of Neural Classifiers.

XF Lin,YS Wu,XQ Ding

1997-01-01

Abstract:In this paper the mechanism behind the combination of multiple neural classifiers based on the same features is explored. In order to improve the accuracy of the estimated posteriori probabilities, linear regression is proposed to combine multiple neural classifiers. In addition, an effective method is introduced to map the outputs of neural networks to posteriori probabilities. The practice in handwritten numeral recognition has proved the effectiveness of the proposed method.
Combining classifiers: A theoretical framework

J. Kittler

DOI: https://doi.org/10.1007/bf01238023

IF: 2.307

1998-03-01

Pattern Analysis and Applications

Abstract:The problem of classifier combination is considered in the context of the two main fusion scenarios: fusion of opinions based on identical and on distinct representations. We develop a theoretical framework for classifier combination for these two scenarios. For multiple experts using distinct representations we argue that many existing schemes such as the product rule, sum rule, min rule, max rule, majority voting, and weighted combination, can be considered as special cases of compound classification. We then consider the effect of classifier combination in the case of multiple experts using a shared representation where the aim of fusion is to obtain a better estimate of the appropriatea posteriori class probabilities. We also show that the two theoretical frameworks can be used for devising fusion strategies when the individual experts use features some of which are shared and the remaining ones distinct. We show that in both cases (distinct and shared representations), the expert fusion involves the computation of a linear or nonlinear function of thea posteriori class probabilities estimated by the individual experts. Classifier combination can therefore be viewed as a multistage classification process whereby thea posteriori class probabilities generated by the individual classifiers are considered as features for a second stage classification scheme. Most importantly, when the linear or nonlinear combination functions are obtained by training, the distinctions between the two scenarios fade away, and one can view classifier fusion in a unified way.

computer science, artificial intelligence
Linear and Order Statistics Combiners for Pattern Classification

Kagan Tumer,Joydeep Ghosh

DOI: https://doi.org/10.48550/arXiv.cs/9905012

1999-05-20

Neural and Evolutionary Computing

Abstract:Several researchers have experimentally shown that substantial improvements can be obtained in difficult pattern recognition problems by combining or integrating the outputs of multiple classifiers. This chapter provides an analytical framework to quantify the improvements in classification results due to combining. The results apply to both linear combiners and order statistics combiners. We first show that to a first order approximation, the error rate obtained over and above the Bayes error rate, is directly proportional to the variance of the actual decision boundaries around the Bayes optimum boundary. Combining classifiers in output space reduces this variance, and hence reduces the "added" error. If N unbiased classifiers are combined by simple averaging, the added error rate can be reduced by a factor of N if the individual errors in approximating the decision boundaries are uncorrelated. Expressions are then derived for linear combiners which are biased or correlated, and the effect of output correlations on ensemble performance is quantified. For order statistics based non-linear combiners, we derive expressions that indicate how much the median, the maximum and in general the ith order statistic can improve classifier performance. The analysis presented here facilitates the understanding of the relationships among error rates, classifier boundary distributions, and combining in output space. Experimental results on several public domain data sets are provided to illustrate the benefits of combining and to support the analytical results.
Optimal partition of feature using Bayesian classifier

Sanjay Vishwakarma,Srinjoy Ganguly

2024-12-08

Abstract:The Naive Bayesian classifier is a popular classification method employing the Bayesian paradigm. The concept of having conditional dependence among input variables sounds good in theory but can lead to a majority vote style behaviour. Achieving conditional independence is often difficult, and they introduce decision biases in the estimates. In Naive Bayes, certain features are called independent features as they have no conditional correlation or dependency when predicting a classification. In this paper, we focus on the optimal partition of features by proposing a novel technique called the Comonotone-Independence Classifier (CIBer) which is able to overcome the challenges posed by the Naive Bayes method. For different datasets, we clearly demonstrate the efficacy of our technique, where we achieve lower error rates and higher or equivalent accuracy compared to models such as Random Forests and XGBoost.

Machine Learning,Artificial Intelligence
Optimal Portfolio Choice under Decision-Based Model Combinations

Davide Pettenuzzo,Francesco Ravazzolo

DOI: https://doi.org/10.2139/ssrn.2516950

2014-01-01

SSRN Electronic Journal

Abstract:We propose a density combination approach featuring combination weights that depend on the past forecast performance of the individual models entering the combination through a utility-based objective function. We apply this model combination scheme to forecast stock returns, both at the aggregate level and by industry, and investigate its forecasting performance relative to a host of existing combination methods, both within the class of linear and time-varying coefficients, stochastic volatility models. Overall, we find that our combination scheme produces markedly more accurate predictions than the existing alternatives, both in terms of statistical and economic measures of out-of-sample predictability.
A general procedure for combining binary classifiers and its performance analysis

Hai Zhao,Bao-Liang Lu

DOI: https://doi.org/10.1007/11539087_36

2005-01-01

Abstract:A general procedure for combining binary classifiers for multiclass classification problems with one-against-one decomposition policy is presented in this paper. Two existing schemes, namely the min-max combination and the most-winning combination, may be regarded as its two special cases. We show that the accuracy of the combination procedure will increase and time complexity will decrease as its main parameter increases under a proposed selection algorithm. The experiments verify our main results, and our theoretical analysis gives a valuable criterion for choosing different schemes of combining binary classifiers.
Combination of Multiple Classifier Using Feature Space Partitioning

Xia Yingju,Hou Cuiqin,Sun Jun

DOI: https://doi.org/10.1007/978-981-10-3002-4_50

2016-01-01

Abstract:Combination of Multiple Classifier has been consider as the approach of improving the classification performance. The popular diversification approach named local specialization is based on the simultaneous partitioning of the feature space and an assignment of a compound classifier to each of the sub-space. This paper presents a novel feature space partitioning algorithm for the combination of multiple classifier. The proposed method uses pairwise measure to get the diversity between classifiers and selects the complementary classifiers to get the pseudo labels. Based on the pseudo labels, it splits the feature space into constituents and selects the best classifier committee from the pool of available classifiers. The partitioning and selection are taken place simultaneously as part of a compound optimization process aimed at maximizing system performance. Evolutionary methods are used to find the optimal solution. The experimental results show the effectiveness and efficiency of the proposed method.
A method of combining multiple probabilistic classifiers through soft competition on different feature sets

Ke Chen,Huisheng Chi

DOI: https://doi.org/10.1016/S0925-2312(98)00019-8

IF: 6

1998-01-01

Neurocomputing

Abstract:A novel method is proposed for combining multiple probabilistic classifiers on different feature sets. In order to achieve the improved classification performance, a generalized finite mixture model is proposed as a linear combination scheme and implemented based on radial basis function networks. In the linear combination scheme, soft competition on different feature sets is adopted as an automatic feature rank mechanism so that different feature sets can be always simultaneously used in an optimal way to determine linear combination weights. For training the linear combination scheme, a learning algorithm is developed based on Expectation–Maximization (EM) algorithm. The proposed method has been applied to a typical real-world problem, viz., speaker identification, in which different feature sets often need consideration simultaneously for robustness. Simulation results show that the proposed method yields good performance in speaker identification.
Classifier Combination Based on Active Learning

X Yi,ZB Kou,CS Zhang

DOI: https://doi.org/10.1109/icpr.2004.1334054

2004-01-01

Abstract:In this paper, we propose classifier combination based on active learning, which deals with the design of classifier combination systems as training a combiner at the aggregation level and introduces SVM active learning into the design of this multi-category decision combiner. This algorithm greatly reduces the number of labeled data the classifier system needs in order to achieve satisfactory performance. This algorithm consists of two main steps: firstly, designing and training first level classifiers which can output posterior probability vectors as the input of the second level combiner, secondly, designing second level combiner based on SVM active learning and classifying testing samples with this combiner. Experiments on standard database show that our algorithm performs better than current classifier combination rules when considering both labeling cost and classification accuracy.
Feature Combination Via Clustering

Jian Hou,Huijun Gao,Xuelong Li

DOI: https://doi.org/10.1109/tnnls.2016.2645883

IF: 14.255

2018-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:In image classification, feature combination is often used to combine the merits of multiple complementary features and improve the classification accuracy compared with one single feature. Existing feature combination algorithms, e.g., multiple kernel learning, usually determine the weights of features based on the optimization with respect to some classifier-dependent objective function. These algorithms are often computationally expensive, and in some cases are found to perform no better than simple baselines. In this paper, we solve the feature combination problem from a totally different perspective. Our algorithm is based on the simple idea of combining only base kernels suitable to be combined. Since the very aim of feature combination is to obtain the highest possible classification accuracy, we measure the combination suitableness of two base kernels by the maximum possible cross-validation accuracy of their combined kernel. By regarding the pairwise suitableness as the kernel adjacency, we obtain a weighted graph of all base kernels and find that the base kernels suitable to be combined correspond to a cluster in the graph. We then use the dominant sets algorithm to find the cluster and determine the weights of base kernels automatically. In this way, we transform the kernel combination problem into a clustering one. Our algorithm can be implemented in parallel easily and the running time can be adjusted based on available memory to a large extent. In experiments on several data sets, our algorithm generates comparable classification accuracy with the state of the art.
Optimal Symmetries in Binary Classification

Vishal S. Ngairangbam,Michael Spannowsky

2024-08-17

Abstract:We explore the role of group symmetries in binary classification tasks, presenting a novel framework that leverages the principles of Neyman-Pearson optimality. Contrary to the common intuition that larger symmetry groups lead to improved classification performance, our findings show that selecting the appropriate group symmetries is crucial for optimising generalisation and sample efficiency. We develop a theoretical foundation for designing group equivariant neural networks that align the choice of symmetries with the underlying probability distributions of the data. Our approach provides a unified methodology for improving classification accuracy across a broad range of applications by carefully tailoring the symmetry group to the specific characteristics of the problem. Theoretical analysis and experimental results demonstrate that optimal classification performance is not always associated with the largest equivariant groups possible in the domain, even when the likelihood ratio is invariant under one of its proper subgroups, but rather with those subgroups themselves. This work offers insights and practical guidelines for constructing more effective group equivariant architectures in diverse machine-learning contexts.

Machine Learning,Artificial Intelligence,Data Analysis, Statistics and Probability
Optimal Bayesian design for model discrimination via classification

Markus Hainy,David J. Price,Olivier Restif,Christopher Drovandi

DOI: https://doi.org/10.1007/s11222-022-10078-2

2022-04-06

Abstract:Performing optimal Bayesian design for discriminating between competing models is computationally intensive as it involves estimating posterior model probabilities for thousands of simulated datasets. This issue is compounded further when the likelihood functions for the rival models are computationally expensive. A new approach using supervised classification methods is developed to perform Bayesian optimal model discrimination design. This approach requires considerably fewer simulations from the candidate models than previous approaches using approximate Bayesian computation. Further, it is easy to assess the performance of the optimal design through the misclassification error rate. The approach is particularly useful in the presence of models with intractable likelihoods but can also provide computational advantages when the likelihoods are manageable.

Computation
Optimal Combination of Feature Weight Learning and Classification Based on Local Approximation

Hongmin Cai,Michael Ng

DOI: https://doi.org/10.1007/978-3-642-34679-8_9

2023-01-01

Abstract:Currently, most feature weights estimation methods are independent on the classification algorithms. The combination of discriminant analysis and classifiers for effective pattern classification remains heuristic. The present study address the topics of learning of feature weights by using a recently reported classification algorithm, K-Local Hyperplane Distance Nearest Neighbor (HKNN) [18], in which the data is modeled as embedded in a linear hyperplane. Motivated by the encouraging performance of the Learning Discriminative Projections and Prototypes, the feature weights are estimated by minimizing the classifier leave-one-out cross validation error of HKNN. Approximated explicit solution is obtained to give feature estimation. Therefore, the feature weighting and classification are perfectly matched. The performance of the combinational model is extensively assessed via experiments on both synthetic and benchmark datasets. The superior results demonstrate that the method is competitive compared with some state-of-art models.
Ensemble Classification Based On Random Linear Base Classifiers

Qi Xiao,Zhengdao Wang

DOI: https://doi.org/10.1109/ICASSP.2017.7952648

2017-01-01

Abstract:We propose a simple ensemble classification algorithm, which employs a set of N randomly generated linear classifiers, followed by a selection process based on the performance of these classifiers on the whole set of training data. The top n performers are then linearly combined to form the final classifier. We analyze the VC dimension of the resulting hypothesis set from such a construction procedure, and show that it can be controlled by choosing the parameters N and n. The proposed algorithm enjoys low computational complexity, and for the MNIST dataset and several UCI datasets that we tested, the algorithm compares favorably in generalization error rate or running time to competing algorithms including Random Kitchen Sinks and AdaBoost.
Optimal linear ensemble of binary classifiers

Mehmet Eren Ahsen,Robert Vogel,Gustavo Stolovitzky

DOI: https://doi.org/10.1093/bioadv/vbae093

2024-06-25

Abstract:Motivation: The integration of vast, complex biological data with computational models offers profound insights and predictive accuracy. Yet, such models face challenges: poor generalization and limited labeled data. Results: To overcome these difficulties in binary classification tasks, we developed the Method for Optimal Classification by Aggregation (MOCA) algorithm, which addresses the problem of generalization by virtue of being an ensemble learning method and can be used in problems with limited or no labeled data. We developed both an unsupervised (uMOCA) and a supervised (sMOCA) variant of MOCA. For uMOCA, we show how to infer the MOCA weights in an unsupervised way, which are optimal under the assumption of class-conditioned independent classifier predictions. When it is possible to use labels, sMOCA uses empirically computed MOCA weights. We demonstrate the performance of uMOCA and sMOCA using simulated data as well as actual data previously used in Dialogue on Reverse Engineering and Methods (DREAM) challenges. We also propose an application of sMOCA for transfer learning where we use pre-trained computational models from a domain where labeled data are abundant and apply them to a different domain with less abundant labeled data. Availability and implementation: GitHub repository, https://github.com/robert-vogel/moca.

Optimal Linear Combination of Classifiers

Optimal Linear Combination Using Decision Reliability Of Individual Classifiers

Optimal Classifier Fusion in a Non-Bayesian Probabilistic Framework

Combination of linear classifiers using score function -- analysis of possible combination strategies

Decision combination in multiple classifier systems

Methods of Linear Combination Based on Different Features

Linear Regression Based Combination of Neural Classifiers.

Combining classifiers: A theoretical framework

Linear and Order Statistics Combiners for Pattern Classification

Optimal partition of feature using Bayesian classifier

Optimal Portfolio Choice under Decision-Based Model Combinations

A general procedure for combining binary classifiers and its performance analysis

Combination of Multiple Classifier Using Feature Space Partitioning

A method of combining multiple probabilistic classifiers through soft competition on different feature sets

Classifier Combination Based on Active Learning

Feature Combination Via Clustering

Optimal Symmetries in Binary Classification

Optimal Bayesian design for model discrimination via classification

Optimal Combination of Feature Weight Learning and Classification Based on Local Approximation

Ensemble Classification Based On Random Linear Base Classifiers

Optimal linear ensemble of binary classifiers