Stylized Neural Painting

Zhengxia Zou,Tianyang Shi,Shuang Qiu,Yi Yuan,Zhenwei Shi

DOI: https://doi.org/10.48550/arXiv.2011.08114

2020-11-17

Abstract:This paper proposes an image-to-painting translation method that generates vivid and realistic painting artworks with controllable styles. Different from previous image-to-image translation methods that formulate the translation as pixel-wise prediction, we deal with such an artistic creation process in a vectorized environment and produce a sequence of physically meaningful stroke parameters that can be further used for rendering. Since a typical vector render is not differentiable, we design a novel neural renderer which imitates the behavior of the vector renderer and then frame the stroke prediction as a parameter searching process that maximizes the similarity between the input and the rendering output. We explored the zero-gradient problem on parameter searching and propose to solve this problem from an optimal transportation perspective. We also show that previous neural renderers have a parameter coupling problem and we re-design the rendering network with a rasterization network and a shading network that better handles the disentanglement of shape and color. Experiments show that the paintings generated by our method have a high degree of fidelity in both global appearance and local textures. Our method can be also jointly optimized with neural style transfer that further transfers visual style from other images. Our code and animated results are available at \url{<a class="link-external link-https" href="https://jiupinjia.github.io/neuralpainter/" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to generate vivid and realistic painting artworks with controllable styles. Specifically, the author proposes a method based on stroke rendering, which can generate high - fidelity painting works with a high degree of authenticity not only in global appearance but also in local texture. Different from previous image - to - image conversion methods, which usually regard the conversion process as pixel - level prediction, the artistic creation process proposed in this paper is carried out in a vectorized environment, generating a series of physically meaningful stroke parameters that can be used for further rendering. In addition, this method can also be jointly optimized with neural style transfer to further transfer visual styles from other images.

To achieve this goal, the paper addresses the following key issues:

1. **Stroke Parameter Search**: Since typical vector rendering is non - differentiable, the author designs a new neural renderer to mimic the behavior of vector rendering and redefines stroke prediction as a parameter search process, aiming to maximize the similarity between the input image and the rendered output.

2. **Zero - Gradient Problem**: The zero - gradient problem is encountered during the parameter search process. The author proposes a solution from the perspective of optimal transport, introducing a differentiable transport loss, which improves stroke convergence and painting results.

3. **Neural Renderer Architecture**: To solve the parameter coupling problem in previous neural renderers, the author redesigns the rendering network, decomposing it into a rasterization network and a shading network, which better handles the decoupling of shape and color and improves the quality of rendering.

Through these innovations, the method proposed in the paper can generate vivid paintings on various real - life images and photos, showing a high degree of realism and artistic sense, both in terms of global visual appearance and local texture fidelity.