Color Hint-guided Ink Wash Painting Colorization with Ink Style Prediction Mechanism

Yao Zeng,Xiaoyu Liu,Yijun Wang,Junsong Zhang

DOI: https://doi.org/10.1145/3657637

2024-04-11

ACM Transactions on Applied Perception

Abstract:We propose an end-to-end generative adversarial network that allows for controllable ink wash painting generation from sketches by specifying the colors via color hints. To the best of our knowledge, this is the first study for interactive Chinese ink wash painting colorization from sketches. To help our network understand the ink style and artistic conception, we introduced an ink style prediction mechanism for our discriminator, which enables the discriminator to accurately predict the style with the help of a pre-trained style encoder. We also designed our generator to receive multi-scale feature information from the feature pyramid network for detail reconstruction of ink wash painting. Experimental results and user study show that ink wash paintings generated by our network have higher realism and richer artistic conception than existing image generation methods.

computer science, software engineering

What problem does this paper attempt to address?

The problem this paper attempts to address is the controllable generation of Chinese ink paintings by specifying color hints, particularly the transformation from sketches to colored paintings with an ink style. Specifically, the paper proposes an end-to-end Generative Adversarial Network (GAN) that allows users to control the process of generating ink paintings from sketches through color hints. This is the first study on interactive Chinese ink painting coloring from sketches.

To help the network understand the ink style and artistic conception, the authors introduce an ink style prediction mechanism, enabling the discriminator to accurately predict the style with the aid of a pre-trained style encoder. Additionally, the authors design the generator to receive multi-scale feature information from a feature pyramid network to reconstruct the details of the ink painting. Experimental results and user studies show that the ink paintings generated by this network outperform existing image generation methods in terms of realism and artistic conception.

In summary, the main contributions of this paper include:
1. Proposing a new color hint-based ink painting coloring framework, with the generator designed to receive multi-scale feature information from a feature pyramid network to achieve detailed reconstruction of ink painting details.
2. Introducing an ink style prediction mechanism, embedding a style prediction module in the discriminator, allowing the discriminator to accurately predict the style under the guidance of a pre-trained style encoder, thereby enhancing the generator's ability to produce appealing ink paintings.
3. To facilitate subsequent research, the authors introduce a new dataset "InkPaint" containing 8,063 high-quality Chinese ink paintings, covering five thematic categories: figures, animals, flowers, fruits, and vegetables.