Abstract:Model merging aims to combine multiple expert models into a more capable single model, offering benefits such as reduced storage and serving costs, improved generalization, and support for decentralized model development. Despite its promise, previous studies have primarily focused on merging a few small models. This leaves many unanswered questions about the effect of scaling model size and how it interplays with other key factors -- like the base model quality and number of expert models -- , to affect the merged model's performance. This work systematically evaluates the utility of model merging at scale, examining the impact of these different factors. We experiment with merging fully fine-tuned models using 4 popular merging methods -- Averaging, Task~Arithmetic, Dare, and TIES -- across model sizes ranging from 1B-64B parameters and merging up to 8 different expert models. We evaluate the merged models on both held-in tasks, i.e., the expert's training tasks, and zero-shot generalization to unseen held-out tasks. Our experiments provide several new insights about model merging at scale and the interplay between different factors. First, we find that merging is more effective when experts are created from strong base models, i.e., models with good zero-shot performance. Second, larger models facilitate easier merging. Third merging consistently improves generalization capabilities. Notably, when merging 8 large expert models, the merged models often generalize better compared to the multitask trained models. Fourth, we can better merge more expert models when working with larger models. Fifth, different merging methods behave very similarly at larger scales. Overall, our findings shed light on some interesting properties of model merging while also highlighting some limitations. We hope that this study will serve as a reference point on large-scale merging for upcoming research.

ListMerge: Accelerating Top-k Aggregation Queries over Large Number of Lists.

Efficient sorting, duplicate removal, grouping, and aggregation

Aggregation-Aware Top-k Computation for Full-Text Search

An Optimization Framework for Merging Multiple Result Lists

Parallel Algorithms for Top-k Query Processing

A Time-space Efficient Algorithm for Parallel K-Way In-place Merging Based on Sequence Partitioning and Perfect Shuffle

What Matters for Model Merging at Scale?

Statistical Consistency of Top-k Ranking.

Efficient Top-K Retrieval on Massive Data

Unordered Task-Parallel Augmented Merge Tree Construction

Optimizing top-k retrieval: submodularity analysis and search strategies

Diversifying Top-K Results

Efficient Parallel Top-k Computation Algorithm Using Symmetry Breaking

Exactly mergeable summaries

Efficient Top-K Query Algorithms Using Density Index

Avoiding monotony: improving the diversity of recommendation lists

MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation

Top-k Query Based on Map Reduce

Listwise Approach For Rank Aggregation In Crowdsourcing

Enhanced Merge Sort- A New Approach to the Merging Process

A New Hierarchical Ranking Aggregation Method