Diversified Arbitrary Style Transfer via Deep Feature Perturbation

Zhizhong Wang,Lei Zhao,Haibo Chen,Lihong Qiu,Qihang Mo,Sihuan Lin,Wei Xing,Dongming Lu

DOI: https://doi.org/10.48550/arXiv.1909.08223

2020-03-20

Abstract:Image style transfer is an underdetermined problem, where a large number of solutions can satisfy the same constraint (the content and style). Although there have been some efforts to improve the diversity of style transfer by introducing an alternative diversity loss, they have restricted generalization, limited diversity and poor scalability. In this paper, we tackle these limitations and propose a simple yet effective method for diversified arbitrary style transfer. The key idea of our method is an operation called deep feature perturbation (DFP), which uses an orthogonal random noise matrix to perturb the deep image feature maps while keeping the original style information unchanged. Our DFP operation can be easily integrated into many existing WCT (whitening and coloring transform)-based methods, and empower them to generate diverse results for arbitrary styles. Experimental results demonstrate that this learning-free and universal method can greatly increase the diversity while maintaining the quality of stylization.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the problem of lack of diversity in image style transfer. Specifically, although existing style transfer methods have made remarkable progress in terms of efficiency, quality, universality, user control, and photo - realism, they often overlook the important aspect of diversity. Many applications (such as artistic creation and creative design) need to satisfy different user preferences, so diversity has become a problem that urgently needs to be solved.

The paper points out that image style transfer is an under - determined problem, that is, there are a large number of solutions that can meet the same content and style constraints. However, existing methods either converge to similar local optimal solutions during the optimization process, or the feed - forward network can only produce fixed results for fixed inputs, resulting in a lack of meaningful variation in the generated results. Although some methods attempt to improve diversity by introducing diversity loss functions, these methods have problems such as limited generalization ability, limited diversity, and poor scalability.

To solve these problems, the author proposes a simple and effective method - Deep Feature Perturbation (DFP). The core idea of this method is to perturb the deep - layer image feature maps using an orthogonal random noise matrix while keeping the original style information unchanged. In this way, the DFP operation can be easily integrated into existing methods based on WCT (Whitening and Coloring Transform) to generate diverse arbitrary - style transfer results. Experimental results show that this method without an additional learning process can significantly increase diversity while maintaining the stylized quality.