Abstract:Exemplar-based colorization approaches rely on reference image to provide plausible colors for target gray-scale image. The key and difficulty of exemplar-based colorization is to establish an accurate correspondence between these two images. Previous approaches have attempted to construct such a correspondence but are faced with two obstacles. First, using luminance channels for the calculation of correspondence is inaccurate. Second, the dense correspondence they built introduces wrong matching results and increases the computation burden. To address these two problems, we propose Semantic-Sparse Colorization Network (SSCN) to transfer both the global image style and detailed semantic-related colors to the gray-scale image in a coarse-to-fine manner. Our network can perfectly balance the global and local colors while alleviating the ambiguous matching problem. Experiments show that our method outperforms existing methods in both quantitative and qualitative evaluation and achieves state-of-the-art performance.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to establish an accurate correspondence between grayscale images and color reference images in example - based image coloring tasks. Specifically, the paper points out that there are two main obstacles in current methods: 1. **Inaccurate calculation of correspondence using the luminance channel**: Due to the lack of sufficient semantic information in grayscale images, using the luminance channel to calculate the correspondence will lead to inaccurate results. 2. **Dense correspondence introduces false matches and increases the computational burden**: Dense correspondence not only introduces false matching results but also significantly increases the computational complexity. To solve these problems, the authors propose a new framework named **Semantic - Sparse Colorization Network (SSCN)**, which can transfer the global image style and detailed semantically - related colors in a coarse - to - fine process. Specific contributions include: - **Constructing more accurate correspondence**: By using the correspondence between the coarse coloring results and the reference image, the information gap between the grayscale input and the color reference is reduced, and better performance is achieved in details. - **Sparse attention mechanism**: Enables the model to focus on semantically important regions in the reference image, thereby generating more detailed results while reducing the computational cost. - **New test data set and evaluation metrics**: A new test data set is collected, and new quantitative evaluation metrics are designed to solve the problem of fair comparison. ### Method Overview The SSCN framework contains two auxiliary modules for transferring global and local colors respectively: 1. **Global Color Transfer (GCT)**: - Use the features \( F_I^r \) of the reference image to perform preliminary coloring on the grayscale image \( I_g \) to obtain the coarse coloring result \( I_c \). - Change the feature statistics through the AdaIN operation to transfer the color style of the reference image to the grayscale image. 2. **Local Details Transfer (LDT)**: - Use the features \( F_I^c \) and \( F_I^r \) of the coarse coloring result \( I_c \) and the reference image \( I_r \) to construct more detailed and accurate correspondence. - Through the sparse attention mechanism, select semantically important regions for matching to reduce the interference of irrelevant regions. ### Experimental Results - **Visual comparison**: Compared with existing example - based coloring methods, SSCN shows better performance on multiple test images, especially when dealing with semantically unrelated reference images. - **Quantitative evaluation**: Through the self - augmentation method (using the augmented version of the reference image as a reference), SSCN outperforms other methods in metrics such as PSNR and SSIM. - **User evaluation**: The subjective evaluation results show that users are more inclined to choose the coloring results generated by SSCN. ### Conclusion SSCN effectively solves the correspondence problem in example - based image coloring tasks through the sparse attention mechanism and the phased coloring framework, and achieves the current state - of - the - art performance.

Semantic-Sparse Colorization Network for Deep Exemplar-based Colorization

Lightweight Deep Exemplar Colorization Via Semantic Attention-Guided Laplacian Pyramid

SPColor: Semantic Prior Guided Exemplar-based Image Colorization

Exemplar-Based Sketch Colorization with Cross-Domain Dense Semantic Correspondence

Deep Exemplar-Based Color Transfer for 3D Model

Deep Exemplar-based Colorization

Spnet: Semantic Preserving Network with Semantic Constraint and Non-Semantic Calibration for Color Constancy

Semantic-driven Colorization

Gray2ColorNet: Transfer More Colors from Reference Image

Unsupervised Deep Exemplar Colorization Via Pyramid Dual Non-Local Attention.

Improving reference-based image colorization for line arts via feature aggregation and contrastive learning

Self-driven Dual-path Learning for Reference-based Line Art Colorization under Limited Data

SCSNet: an Efficient Paradigm for Learning Simultaneously Image Colorization and Super-resolution

Stylization-Based Architecture for Fast Deep Exemplar Colorization

Globally and Locally Semantic Colorization via Exemplar-Based Broad-GAN

Semantic-Aware Gated Fusion Network For Interactive Colorization

Adversarial Edge-Aware Image Colorization with Semantic Segmentation.

Bidirectional Layout-Semantic-Pixel Joint Decoupling and Embedding Network for Remote Sensing Colorization

Semantic Reconstruction based on RGB Image and Sparse Depth

Exemplar-based video colorization with long-term spatiotemporal dependency

Color Constancy via Multi-Scale Region-Weighed Network Guided by Semantics