STRAT: Image Style Transfer with Region-Aware Transformer

Na Qi,Yezi Li,Rao Fu,Qing Zhu

DOI: https://doi.org/10.1016/j.neucom.2024.129039

IF: 6

2024-01-01

Neurocomputing

Abstract:Style transfer methods render various artistic styles to a natural image through the extraction and transfer of textural features. Existing neural style transfer methods often rely on CNNs to extract image features and tend to suffer from feature leakage and content distortion due to limited receptive fields. Transformer-based style transfer methods outperform CNN-based methods by learning the global information of image through self-attention mechanism. However, local features are ignored and details are lost since the semantic information of images is not taken into account. To address this critical issue, this paper proposes a novel style transfer framework based on region-aware transformer (STRAT). We integrate the CNN based short-range branch with the transformer-based long-range branch to extract both local and non-local features to achieve region-adaptive texture transfer with two region-aware attention modules, respectively. Specifically, we utilize the SNR metric and masks as guide to propose the SNR-guided attention module and mask-guided cross attention module to enable region-varying feature extraction and adaptive texture transfer, respectively. Extensive experimental results demonstrate that our proposed method outperforms the state-of-the-arts methods in terms of subjective and objective results.