Curved-stroke-based Neural Painting and Stylization Through Thin Plate Spline Interpolation
Bohao TANG,Teng HU,Yuzhen DU,Ran YI,Lizhuang MA
DOI: https://doi.org/10.1360/ssi-2023-0194
2023-01-01
Scientia Sinica Informationis
Abstract:In recent years,there have been remarkable advancements in image generation technology.Existing image generation methods mostly generate images by filling pixels,which lacks the stroke-by-stroke drawing process of an artist,resulting in differences in brushstroke details and textures from those found in real artworks.On the contrary,neural painting aims to imitate the stroke-by-stroke drawing process of a human artist with brushstrokes as the basic unit.Most existing neural painting methods simulate real brushstrokes using Bezier curves or stroke templates through affine transformations.However,the lack of texture in Bezier curves and the linear nature of affine transformations indicate that the generated brushstrokes have significant limitations in terms of texture or shape.To better simulate the texture and shape of real brushstrokes,we propose a new curved brushstroke parameter model based on thin plate spline interpolation.By curving and affine-transforming real brushstroke templates in succession,we can generate more realistic and varied brushstroke images.Furthermore,we propose a hierarchical brushstroke optimization method that decomposes the entire image into multiple brushstrokes,from large to small,effectively improving the model's painting ability for both the overall structure and local details of the image.Finally,we extend the proposed method to style transfer,achieving impressive style transfer effects.Qualitative and quantitative experiments demonstrate that our new brushstroke model and optimization method outperform the existing models in both neural painting and stylization tasks.