Abstract:Large language models (LLMs) exhibit complementary strengths in various tasks, motivating the research of LLM ensembling. However, existing work focuses on training an extra reward model or fusion model to select or combine all candidate answers, posing a great challenge to the generalization on unseen data distributions. Besides, prior methods use textual responses as communication media, ignoring the valuable information in the internal representations. In this work, we propose a training-free ensemble framework DeePEn, fusing the informative probability distributions yielded by different LLMs at each decoding step. Unfortunately, the vocabulary discrepancy between heterogeneous LLMs directly makes averaging the distributions unfeasible due to the token misalignment. To address this challenge, DeePEn maps the probability distribution of each model from its own probability space to a universal relative space based on the relative representation theory, and performs aggregation. Next, we devise a search-based inverse transformation to transform the aggregated result back to the probability space of one of the ensembling LLMs (main model), in order to determine the next token. We conduct extensive experiments on ensembles of different number of LLMs, ensembles of LLMs with different architectures, and ensembles between the LLM and the specialist model. Experimental results show that (i) DeePEn achieves consistent improvements across six benchmarks covering subject examination, reasoning, and knowledge, (ii) a well-performing specialist model can benefit from a less effective LLM through distribution fusion, and (iii) DeePEn has complementary strengths with other ensemble methods such as voting.

Interpretability Diversity for Decision-Tree-Initialized Dendritic Neuron Model Ensemble

Structural Diversity for Decision Tree Ensemble Learning

Inherently Interpretable Tree Ensemble Learning

A DNN-Ensemble Method for Error Reduction and Training Data Selection in DNN Based Modeling

Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness

Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity

An Ensemble Classifier Based on Attribute Selection and Diversity Measure

The Diversified Ensemble Neural Network.

Learning to Diversify via Weighted Kernels for Classifier Ensemble

Integration of Classifier Diversity Measures for Feature Selection-Based Classifier Ensemble Reduction.

Efficient Diversity-Driven Ensemble for Deep Neural Networks

Developing parsimonious ensembles using ensemble diversity within a reinforcement learning framework

Diversified deep hierarchical kernel ensemble regression

A New Diversity Measure for Classifier Fusion

Scalable Ensemble Diversification for OOD Generalization and Detection

Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration

Using dual evolutionary search to construct decision tree based ensemble classifier

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

Diversity Regularized Ensemble Pruning.

Remaining useful life estimation based on selective ensemble of deep neural networks with diversity

Enabling Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration.