Unified Concept Editing in Diffusion Models

Rohit Gandikota,Hadas Orgad,Yonatan Belinkov,Joanna Materzyńska,David Bau

2024-10-23

Abstract:Text-to-image models suffer from various safety issues that may limit their suitability for deployment. Previous methods have separately addressed individual issues of bias, copyright, and offensive content in text-to-image models. However, in the real world, all of these issues appear simultaneously in the same model. We present a method that tackles all issues with a single approach. Our method, Unified Concept Editing (UCE), edits the model without training using a closed-form solution, and scales seamlessly to concurrent edits on text-conditional diffusion models. We demonstrate scalable simultaneous debiasing, style erasure, and content moderation by editing text-to-image projections, and we present extensive experiments demonstrating improved efficacy and scalability over prior work. Our code is available at <a class="link-external link-https" href="https://unified.baulab.info" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to simultaneously handle multiple security issues in text - to - image generation models, such as bias, copyright infringement, and generation of inappropriate content, etc. Specifically, the authors propose a unified concept editing method (Unified Concept Editing, UCE), which can solve these problems by modifying model parameters without retraining the model. The UCE method can edit multiple concepts simultaneously, including removing artistic styles, reducing occupational gender and racial biases, and decreasing the possibility of generating inappropriate content. This method not only improves the safety of the model but also maintains the generation quality of unedited concepts.

### Main Contributions
1. **Unified Editing Method**: UCE provides a unified method to solve multiple security problems in text - to - image generation models without having to deal with each problem separately.
2. **Efficient Editing**: The UCE method uses a closed - form solution to modify model parameters and can edit multiple concepts in a short time, which is highly efficient and effective.
3. **Multi - task Editing**: UCE can handle multiple editing tasks simultaneously, such as removing artistic styles, reducing biases, and content moderation, without interfering with each other.
4. **Maintaining Generation Quality**: While editing specific concepts, UCE can minimize the impact on other unedited concepts and maintain the overall generation quality of the model.

### Specific Applications
- **Removing Artistic Styles**: By modifying model parameters, the model is made to no longer generate the styles of specific artists.
- **Reducing Occupational Biases**: By adjusting the model's generation of occupational names, it can represent different genders and races more fairly when generating occupations such as doctors and CEOs.
- **Content Moderation**: By editing the model, the possibility of generating inappropriate content (such as nude images) is reduced.

### Experimental Results
- **Removing Artistic Styles**: Experiments show that the UCE method can maintain high image - generation quality and low interference when removing the styles of multiple artists.
- **Reducing Occupational Biases**: The UCE method performs well in reducing occupational gender and racial biases and can generate more diverse images.
- **Content Moderation**: The UCE method has also achieved good results in reducing the generation of inappropriate content and can effectively reduce the probability of generating inappropriate images.

In conclusion, this paper proposes an efficient and unified method to solve multiple security problems in text - to - image generation models, which has important practical application value.