Abstract:Image harmonization is a critical task in computer vision, which aims to adjust the foreground to make it compatible with the background. Recent works mainly focus on using global transformations (i.e., normalization and color curve rendering) to achieve visual consistency. However, these models ignore local visual consistency and their huge model sizes limit their harmonization ability on edge devices. In this paper, we propose a hierarchical dynamic network (HDNet) to adapt features from local to global view for better feature transformation in efficient image harmonization. Inspired by the success of various dynamic models, local dynamic (LD) module and mask-aware global dynamic (MGD) module are proposed in this paper. Specifically, LD matches local representations between the foreground and background regions based on semantic similarities, then adaptively adjust every foreground local representation according to the appearance of its K-nearest neighbor background regions. In this way, LD can produce more realistic images at a more fine-grained level, and simultaneously enjoy the characteristic of semantic alignment. The MGD effectively applies distinct convolution to the foreground and background, learning the representations of foreground and background regions as well as their correlations to the global harmonization, facilitating local visual consistency for the images much more efficiently. Experimental results demonstrate that the proposed HDNet significantly reduces the total model parameters by more than 80% compared to previous methods, while still attaining state-of-the-art performance on the popular iHarmony4 dataset. Additionally, we introduced a lightweight version of HDNet, i.e., HDNet-lite, which has only 0.65MB parameters, yet it still achieve competitive performance. Our code is avaliable at https://github.com/chenhaoxing/HDNet.

Image Harmonization Dataset Iharmony4: HCOCO, HAdobe5k, HFlickr, and Hday2night

DoveNet: Deep Image Harmonization via Domain Verification

CcHarmony: Color-checker based Image Harmonization Dataset

Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation

SSH: A Self-Supervised Framework for Image Harmonization

Intrinsic Image Harmonization

NeSF: Neural Shading Field for Image Harmonization

Deep Image-based Illumination Harmonization

HYouTube: Video Harmonization Dataset

FRIH: Fine-grained Region-aware Image Harmonization

Foreground Harmonization and Shadow Generation for Composite Image

Deep Image Harmonization in Dual Color Spaces

Deep Image Harmonization with Learnable Augmentation

HarmonyIQA: Pioneering Benchmark and Model for Image Harmonization Quality Assessment

Retrieval Augmented Image Harmonization

Diverse Image Harmonization

Referring Image Harmonization

Hierarchical Dynamic Image Harmonization

Deep Video Harmonization with Color Mapping Consistency

Image harmonization with Simple Hybrid CNN-Transformer Network

DiffHarmony++: Enhancing Image Harmonization with Harmony-VAE and Inverse Harmonization Model