Abstract:Combining the predictions of multiple trained models through ensembling is generally a good way to improve accuracy by leveraging the different learned features of the models, however it comes with high computational and storage costs. Model fusion, the act of merging multiple models into one by combining their parameters reduces these costs but doesn't work as well in practice. Indeed, neural network loss landscapes are high-dimensional and non-convex and the minima found through learning are typically separated by high loss barriers. Numerous recent works have been focused on finding permutations matching one network features to the features of a second one, lowering the loss barrier on the linear path between them in parameter space. However, permutations are restrictive since they assume a one-to-one mapping between the different models' neurons exists. We propose a new model merging algorithm, CCA Merge, which is based on Canonical Correlation Analysis and aims to maximize the correlations between linear combinations of the model features. We show that our alignment method leads to better performances than past methods when averaging models trained on the same, or differing data splits. We also extend this analysis into the harder setting where more than 2 models are merged, and we find that CCA Merge works significantly better than past methods. Our code is publicly available at <a class="link-external link-https" href="https://github.com/shoroi/align-n-merge" rel="external noopener nofollow">this https URL</a>

Using Correspondence Analysis to Combine Classifiers

Combining Multiple K-Nearest Neighbor Classifiers Using Different Distance Functions

Multiple Classifiers Combination Based on Specialists' FIelds

Decision by Maximum of Posterior Probability Average with Weights: a Method of Multiple Classifiers Combination

Combining Classifiers and Learning Mixture-of-Experts

Combining Classifiers Based on Complex Network

A Classifier Ensemble Model and Its Applications

Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis

A New Multiple Classifiers Combination Algorithm

An Empirical Study of a Linear Regression Combiner on Multi-class Data Sets

New Model of Combining Multiple Classifiers

Model Combination In The Multiple-Data-Batches Scenario

Combination of Heterogeneous Multiple Classifiers Based on Evidence Theory

On The Use of Nonparametric Neighborhood Classification Rules in Multiple Classifier Combination

Combining Different Classifiers in Educational Data Mining

A new probability adjustment method for combining conflicting evidences: application in classifier combination

Linear Regression Based Combination of Neural Classifiers.

Learning from Batched Data: Model Combination Versus Data Combination

Combining Classifiers in Software Quality Prediction: A Neural Network Approach

A method of combining multiple probabilistic classifiers through soft competition on different feature sets

PLeaS -- Merging Models with Permutations and Least Squares