Abstract:Most of the existing image style transfer algorithms transfer the whole image style as a whole. Style feature is a set of correlation matrix based on style image, namely Gram matrix. Each matrix is a global description of the style image. This kind of methods can perform well in the insensitive semantic scenes (such as the style transfer between landscape photos), but in the sensitive semantic scenes (such as the style transfer between portrait photos), the problem of semantic mismatch will be highlighted, such as transferring the background texture of the style image to the foreground of the target image. Although the existing research takes the manually annotated semantic image as an input of the algorithm, and then guides the style transfer based on the semantic information, and finally achieves good results in the style transfer between portraits. But there are still two problems: first, semantic images need to be manually annotated, which costs human resources. In practical applications, large-scale image style transfer is often needed. Second, the details of the synthesized image are fuzzy, and the definition is not enough. We propose an image style transfer algorithm based on semantic segmentation to resolve semantic mismatching in image style transfer. Our algorithm extracts the semantic information of style image and content image automatically through a semantic segmentation network and uses the semantic information to guide the style transfer. Our algorithm builds a semantic segmentation network based on mask R-CNN, introduces semantic information, and then makes style transfer on the patch level, realizes the style transfer between similar objects (consistent semantic information). Experiments on Celeba and Wikiart show that our method could automatically extract the semantic information of style image and content image. Compared with the state-of-art approaches in this field, our method can effectively avoid semantic mismatch in the process of image st-le transfer. That is, it can maintain semantic consistency in the process of style transfer.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the semantic mismatch problem that occurs in image style transfer. Most of the existing image style transfer algorithms transfer the image style as a whole. These methods perform well when dealing with scenarios that are not semantically sensitive (such as style transfer between landscape photos), but when dealing with semantically sensitive scenarios (such as style transfer between portrait photos), semantic mismatch problems will occur, for example, the background texture of the style image is transferred to the foreground of the target image. Although some existing studies have achieved good results in style transfer between portraits by using manually - annotated semantic images as algorithm inputs and guiding style transfer based on semantic information, this method has two main problems: first, semantic images need to be manually annotated, which consumes a large amount of human resources; second, the details of the synthesized image are blurred and the clarity is insufficient. For this reason, the author proposes an image style transfer algorithm based on semantic segmentation (referred to as the SST algorithm for short). This algorithm automatically extracts the semantic information of the style image and the content image by constructing a semantic segmentation network based on Mask R - CNN, and uses this semantic information to guide style transfer, thereby achieving style transfer between similar objects (objects with the same semantic information). Experimental results show that, compared with the advanced methods in the current field, this method can effectively avoid the semantic mismatch problem in the image style transfer process, that is, maintain semantic consistency during the style transfer process.

Image Style Transfer Algorithm Based on Semantic Segmentation

GLStyleNet: Exquisite Style Transfer Combining Global and Local Pyramid Features

Learning Structure-Aware Transformations for Arbitrary Image Style Transfer

Correlation-based and Content-Enhanced Network for Video Style Transfer

Semantic Context-Aware Image Style Transfer

Artistic Style Transfer with Internal-external Learning and Contrastive Learning

Diverse Image Style Transfer Via Invertible Cross-Space Mapping

Optimal Transport of Deep Feature for Image Style Transfer

Image style transfer with collection representation space and semantic-guided reconstruction

Image Neural Style Transfer with Preserving the Salient Regions.

Semantic-related image style transfer with dual-consistency loss.

Automatic Semantic Style Transfer using Deep Convolutional Neural Networks and Soft Masks

Foreground and background separated image style transfer with a single text condition

Aesthetic-Aware Image Style Transfer.

Photographic style transfer

A non-definitive auto-transfer mechanism for arbitrary style transfers

Advanced Deep Learning Techniques for Image Style Transfer: A Survey

A Compositional Transformer Based Autoencoder for Image Style Transfer

Name Your Style: An Arbitrary Artist-aware Image Style Transfer

Any-to-Any Style Transfer: Making Picasso and Da Vinci Collaborate

Respecting Low-Level Components of Content with Skip Connections and Semantic Information in Image Style Transfer