SSH: A Self-Supervised Framework for Image Harmonization
Yifan Jiang,He Zhang,Jianming Zhang,Yilin Wang,Zhe Lin,Kalyan Sunkavalli,Simon Chen,Sohrab Amirghodsi,Sarah Kong,Zhangyang Wang
DOI: https://doi.org/10.48550/arXiv.2108.06805
2021-08-18
Abstract:Image harmonization aims to improve the quality of image compositing by matching the "appearance" (\eg, color tone, brightness and contrast) between foreground and background images. However, collecting large-scale annotated datasets for this task requires complex professional retouching. Instead, we propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. We reformulate the image harmonization problem from a representation fusion perspective, which separately processes the foreground and background examples, to address the background occlusion issue. This framework design allows for a dual data augmentation method, where diverse [foreground, background, pseudo GT] triplets can be generated by cropping an image with perturbations using 3D color lookup tables (LUTs). In addition, we build a real-world harmonization dataset as carefully created by expert users, for evaluation and benchmarking purposes. Our results show that the proposed self-supervised method outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study. Code and dataset are available at \url{<a class="link-external link-https" href="https://github.com/VITA-Group/SSHarmonization" rel="external noopener nofollow">this https URL</a>}.
Computer Vision and Pattern Recognition