Improved Style Transfer by Respecting Inter-layer Correlations

Mao-Chuang Yeh,Shuai Tang

DOI: https://doi.org/10.48550/arXiv.1801.01933

2018-01-06

Abstract:A popular series of style transfer methods apply a style to a content image by controlling mean and covariance of values in early layers of a feature stack. This is insufficient for transferring styles that have strong structure across spatial scales like, e.g., textures where dots lie on long curves. This paper demonstrates that controlling inter-layer correlations yields visible improvements in style transfer methods. We achieve this control by computing cross-layer, rather than within-layer, gram matrices. We find that (a) cross-layer gram matrices are sufficient to control within-layer statistics. Inter-layer correlations improves style transfer and texture synthesis. The paper shows numerous examples on "hard" real style transfer problems (e.g. long scale and hierarchical patterns); (b) a fast approximate style transfer method can control cross-layer gram matrices; (c) we demonstrate that multiplicative, rather than additive style and content loss, results in very good style transfer. Multiplicative loss produces a visible emphasis on boundaries, and means that one hyper-parameter can be eliminated.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to transfer styles with cross - spatial - scale structural features, such as points in textures arranged along long curves, more effectively in style transfer tasks. Traditional style transfer methods achieve the transfer of styles to content images by controlling the mean and covariance of the early layers of neural networks, but this method is insufficient for style transfer tasks that require strong cross - spatial - scale structural features. Specifically, the paper points out:

- **Limitations of traditional methods**: Traditional style transfer methods mainly rely on the Gram matrix within a single layer to match style features, which is not effective in dealing with complex textures or patterns with long - distance correlations. For example, when small elements (such as points) in the style image are organized into larger structures (such as long curves) in a specific way, using only the statistical information within a single layer cannot effectively capture this cross - scale correlation.

- **Proposed new method**: To improve this situation, the paper proposes a new style transfer method, that is, by calculating the cross - layer Gram matrix to control the correlations between different layers. This method can better capture the long - distance structural features in the style image and maintain the integrity of these features during the style transfer process.

- **Specific contributions**:
- It is proved that the cross - layer Gram matrix can effectively control the intra - layer statistical characteristics, thereby achieving significant improvements in style transfer and texture synthesis tasks.
- The advantages of the cross - layer Gram matrix in dealing with "hard" style transfer problems (such as long - scale and hierarchical patterns) are demonstrated.
- A fast approximate style transfer method that can control the cross - layer Gram matrix is proposed.
- It is proved that the multiplicative loss (rather than the additive loss) can produce better results in style transfer, especially in emphasizing content boundaries.

In conclusion, the main goal of this paper is to improve the ability to handle complex structural features in style transfer tasks by introducing the cross - layer Gram matrix and the multiplicative loss mechanism, thereby generating higher - quality stylized images.