Abstract:Style transfer on images has achieved significant advances in recent years, with the deep convolutional neural network (CNN). Directly applying image style transfer algorithms to each frame of a video independently often leads to flickering and unstable results. In this work, we present a self-supervised space-time convolutional neural network (CNN) based method for online video style transfer, named as VTNet, which is end-to-end trained from nearly unlimited unlabeled video data to produce temporally coherent stylized videos in real-time. Specifically, our VTNet transfer the style of a reference image to the source video frames, which is formed by the temporal prediction branch and the stylizing branch. The temporal prediction branch is used to capture discriminative spatiotemporal features for temporal consistency, pretrained in an adversarial manner from unlabeled video data. The stylizing branch is used to transfer the style image to a video frame with the guidance from the temporal prediction branch to ensure temporal consistency. To guide the training of VTNet, we introduce the style-coherence loss net (SCNet), which assembles the content loss, the style loss, and the new designed coherence loss. These losses are computed based on high-level features extracted from a pretrained VGG-16 network. The content loss is used to preserve high-level abstract contents of the input frames, and the style loss introduces new colors and patterns from the style image. Instead of using optical flow to explicitly redress the stylized video frames, we design the coherence loss to make the stylized video inherit the dynamics and motion patterns from the source video to remove temporal flickering. Extensive subjective and objective evaluations on various styles demonstrate that the proposed method achieves favorable results against the state-of-the-arts with high efficiency.

Consistent Panoramic Video Style Transfer Via Temporal-Spatial Cross Perception

Correlation-based and Content-Enhanced Network for Video Style Transfer

GLStyleNet: Exquisite Style Transfer Combining Global and Local Pyramid Features

Incorporating Multiscale Contextual Loss for Image Style Transfer

Learning Structure-Aware Transformations for Arbitrary Image Style Transfer

Artistic Style Transfer with Internal-external Learning and Contrastive Learning

Diverse Image Style Transfer Via Invertible Cross-Space Mapping

Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective.

Preserving Global and Local Temporal Consistency for Arbitrary Video Style Transfer

CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer

Consistent Video Style Transfer Via Compound Regularization.

Semi-Parametric Style Transfer with Multi-Perspective Feature Fusion and Information-Guided Alignment

Neural stereoscopic image style transfer

Stable Video Style Transfer Based on Partial Convolution with Depth-Aware Supervision

Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning

Learning Self-Supervised Space-Time CNN for Fast Video Style Transfer

Real-time Localized Photorealistic Video Style Transfer

Real-Time Neural Style Transfer for Videos

Stereoscopic Neural Style Transfer

Consistent Video Style Transfer Via Relaxation and Regularization