Abstract:Image harmonization aims to adjust the foreground illumination in a composite image to make it harmonious. The existing harmonization methods can only produce one deterministic result for a composite image, ignoring that a composite image could have multiple plausible harmonization results due to multiple plausible reflectances. In this work, we first propose a reflectance-guided harmonization network, which can achieve better performance with the guidance of ground-truth foreground reflectance. Then, we also design a diverse reflectance generation network to predict multiple plausible foreground reflectances, leading to multiple plausible harmonization results. The extensive experiments on the benchmark datasets demonstrate the effectiveness of our method.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve a key problem in image harmonization, that is, how to deal with the uncertainty of foreground reflectance. Specifically:
1. **Limitations of existing methods**:
- Existing image fusion methods can only generate a single definite result for a composite image, ignoring the fact that a composite image may have multiple reasonable fusion results.
- According to the Retinex theory, an image \(I\) can be decomposed into a reflectance map \(A\) and an illumination map \(L\), that is, \(I = A * L\), where \( * \) represents element - wise multiplication. The reflectance map represents the intrinsic color of an object, while the illumination map represents the ambient light. For the foreground in a composite image, its reflectance map is sometimes difficult to determine, so there may be multiple reasonable reflectance maps.
2. **The proposed new method**:
- The author proposes a reflectance - guided harmonization network, which can better adjust the foreground illumination when given the real foreground reflectance map, making the fusion result closer to the real situation.
- At the same time, the author also designs a diverse reflectance generation network to predict multiple reasonable foreground reflectance maps, thus generating multiple reasonable fusion results.
3. **Challenges in practical applications**:
- In practical applications, the original foreground image \(I_o\) is usually unavailable, so the real foreground reflectance map \(A_{gt}\) cannot be directly obtained. For this reason, the author designs a foreground - only diverse reflectance generation network, which generates multiple reasonable foreground reflectance maps by sampling random vectors.
### Main contributions of the paper
- **Explore diverse image fusion**: Consider the uncertainty of foreground reflectance and explore the problem of diverse image fusion.
- **Propose a reflectance - guided fusion network**: Reveal that the real foreground reflectance map can assist in generating fusion images closer to the real results.
- **Design a diverse reflectance generation network**: Be able to generate diverse and reasonable foreground reflectance maps, and then generate multiple reasonable fusion results.
Through these methods, the paper solves the problem that existing image fusion methods can only generate a single definite result, and demonstrates its effectiveness on different datasets.