Semi-Parametric Style Transfer with Multi-Perspective Feature Fusion and Information-Guided Alignment

Tianlong Zhang,Jing Lv,Ming Yang

DOI: https://doi.org/10.1145/3652583.3658027

2024-01-01

Abstract:The goal of style transfer is to render images with the attribute dependencies of style images while maintaining the original content structure. Some recent works mainly extract the statistical information of feature maps to match the target style, but the key challenge is that the captured feature representations are too homogeneous, and cross-domain mutual exclusion may occur in the alignment process, resulting in information loss, mismatch, planarization, and so on. To this end, we propose a semi-parametric framework based on multi-perspective feature fusion and guided alignment (SPMPFA), and a backtracking loss function for content maintenance. The SPMPFA and the backgracking loss work together to capture rich presentation information while maintaining structure, thus achieving style consistency between similar fine-grained semantics and global style hierarchy. Specifically, we first use adaptive aggregation and mapping Transformer (AMTransformer) to build a cross-domain graph carrying location information inside the module and use information based on weight aggregation as associated features to guide the style alignment trend. Then, we use the feature fusion strategy to adaptively fuse the heterogeneous representation information. Finally, the content structure is maintained to the maximum extent by using backtracking loss. Qualitative and quantitative experiments demonstrate the effectiveness of our work compared to other style transfer tasks.