Image Harmonization with Diffusion Model

Jiajie Li,Jian Wang,Chen Wang,Jinjun Xiong

2023-06-18

Abstract:Image composition in image editing involves merging a foreground image with a background image to create a composite. Inconsistent lighting conditions between the foreground and background often result in unrealistic composites. Image harmonization addresses this challenge by adjusting illumination and color to achieve visually appealing and consistent outputs. In this paper, we present a novel approach for image harmonization by leveraging diffusion models. We conduct a comparative analysis of two conditional diffusion models, namely Classifier-Guidance and Classifier-Free. Our focus is on addressing the challenge of adjusting illumination and color in foreground images to create visually appealing outputs that seamlessly blend with the background. Through this research, we establish a solid groundwork for future investigations in the realm of diffusion model-based image harmonization.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the sense of unreality caused by inconsistent lighting conditions when synthesizing the foreground image with the background image in image editing. Specifically, Image Harmonization aims to adjust the appearance of the foreground image to make it consistent with the background image in terms of lighting and color, thereby generating natural and realistic composite images. Traditional methods usually use color transfer techniques to match the color statistics of the foreground and background, but these methods are often unable to handle complex scenes and are prone to visual flaws. Although existing deep - learning methods have improved the quality of harmonization to a certain extent, they are usually limited by the supervised learning setting and may lose details.

To overcome these problems, this paper proposes a new method based on Diffusion Models for image harmonization. By comparing two conditional diffusion models - Classifier - Guidance and Classifier - Free, the paper explores how to achieve seamless fusion of the foreground image and the background image in terms of lighting and color while keeping the image structure and semantics unchanged. In addition, the paper also proposes a selective color transfer method to ensure the appearance consistency of the generated image, and verifies the effectiveness of the proposed method through experiments, demonstrating its superior performance in the image harmonization task.