Model Merging and Safety Alignment: One Bad Model Spoils the Bunch

Hasan Abed Al Kader Hammoud,Umberto Michieli,Fabio Pizzati,Philip Torr,Adel Bibi,Bernard Ghanem,Mete Ozay

2024-06-21

Abstract:Merging Large Language Models (LLMs) is a cost-effective technique for combining multiple expert LLMs into a single versatile model, retaining the expertise of the original ones. However, current approaches often overlook the importance of safety alignment during merging, leading to highly misaligned models. This work investigates the effects of model merging on alignment. We evaluate several popular model merging techniques, demonstrating that existing methods do not only transfer domain expertise but also propagate misalignment. We propose a simple two-step approach to address this problem: (i) generating synthetic safety and domain-specific data, and (ii) incorporating these generated data into the optimization process of existing data-aware model merging techniques. This allows us to treat alignment as a skill that can be maximized in the resulting merged LLM. Our experiments illustrate the effectiveness of integrating alignment-related data during merging, resulting in models that excel in both domain expertise and alignment.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

This paper mainly discusses the security alignment issues that arise during the merging process of Large Language Models (LLMs). Current methods often overlook the importance of secure alignment when merging multiple expert models, resulting in significant alignment problems in the merged model. It was found that simply merging expert models containing one or more insecurely aligned models can make the merged model also insecure.

To address this issue, the paper proposes a two-step approach: first, generating synthetic secure and domain-specific data, and then incorporating these generated data into the optimization process of existing data-aware model merging techniques. This allows alignment to be integrated as a maximizable capability into the merged LLM. Experimental results show that by integrating alignment-related data in the merging process, models that excel in domain expertise and have good alignment can be created.

The paper also compares different model merging techniques and demonstrates the effectiveness of security-aware model merging in preserving model alignment properties. In addition, the paper mentions existing model alignment, model merging techniques, and alignment evaluation methods, emphasizing the necessity of considering security impacts in the merging process.

In summary, the problem that the paper aims to address is how to ensure secure alignment of models when merging Large Language Models, in order to avoid the propagation of insecure content and maintain or improve the models' domain expertise.