Abstract:Model merging aims to combine multiple expert models into a more capable single model, offering benefits such as reduced storage and serving costs, improved generalization, and support for decentralized model development. Despite its promise, previous studies have primarily focused on merging a few small models. This leaves many unanswered questions about the effect of scaling model size and how it interplays with other key factors -- like the base model quality and number of expert models -- , to affect the merged model's performance. This work systematically evaluates the utility of model merging at scale, examining the impact of these different factors. We experiment with merging fully fine-tuned models using 4 popular merging methods -- Averaging, Task~Arithmetic, Dare, and TIES -- across model sizes ranging from 1B-64B parameters and merging up to 8 different expert models. We evaluate the merged models on both held-in tasks, i.e., the expert's training tasks, and zero-shot generalization to unseen held-out tasks. Our experiments provide several new insights about model merging at scale and the interplay between different factors. First, we find that merging is more effective when experts are created from strong base models, i.e., models with good zero-shot performance. Second, larger models facilitate easier merging. Third merging consistently improves generalization capabilities. Notably, when merging 8 large expert models, the merged models often generalize better compared to the multitask trained models. Fourth, we can better merge more expert models when working with larger models. Fifth, different merging methods behave very similarly at larger scales. Overall, our findings shed light on some interesting properties of model merging while also highlighting some limitations. We hope that this study will serve as a reference point on large-scale merging for upcoming research.

On the Average Behavior of Set Merging Algorithms (extended Abstract)

Dynamic programming based algorithms for set multicover and multiset multicover problems

Note on the union-closed sets conjecture and Reimer's average set size theorem

Homogeneous structures in subset sums and non-averaging sets

Accelerating Set Intersections over Graphs by Reducing-Merging

Improved Affine Partition Algorithm for Compile-Time and Runtime Performance.

On the number of sets with small sumset

A General Merging Algorithm Based On Object Marking

Sumsets, 3SUM, Subset Sum: Now for Real!

Optimization of Merge Policy in AsterixDB Big Data Management System.

Enhancing Precision of Structured Merge by Proper Tree Matching

ListMerge: Accelerating Top-k Aggregation Queries over Large Number of Lists.

Equal-Subset-Sum Faster Than the Meet-in-the-Middle

Enhanced Merge Sort- A New Approach to the Merging Process

What Matters for Model Merging at Scale?

A Piecewise Approach for the Analysis of Exact Algorithms

Optimal Constructions of Hybrid Algorithms

On the cohesion and separability of average-link for hierarchical agglomerative clustering

Pareto Sums of Pareto Sets: Lower Bounds and Algorithms

Dynamic Privacy-Aware Collaborative Schemes for Average Computation: A Multi-Time Reporting Case

Subdivision schemes for compact sets with a new definition of set average