Abstract:Video-to-Video synthesis (Vid2Vid) gains remarkable performance in generating a photo-realistic video from a sequence of semantic maps, such as segmentation, sketch and pose. However, this pipeline is heavily limited to high computational cost and long inference latency, mainly attributed to two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have been significantly reduced via more efficient network architectures. Existing methods mainly focus on slimming network architectures but ignore the size of the sequential data stream. Moreover, due to the lack of temporal coherence, image-based compression is not sufficient for the compression of the video task. In this paper, we present a spatial-temporal hybrid distillation compression framework, Fast-Vid2Vid++, which focuses on knowledge distillation of the teacher network and the data stream of generative models on both space and time. Fast-Vid2Vid++ makes the first attempt at time dimension to transfer hierarchical features and time coherence knowledge to reduce computational resources and accelerate inference. Specifically, we compress the data stream spatially and reduce the temporal redundancy. We distill the knowledge of the hierarchical features and the final response from the teacher network to the student network in high-resolution and full-time domains. We transfer the long-term dependencies of the features and video frames to the student model. After the proposed spatial-temporal hybrid knowledge distillation (Spatial-Temporal-HKD), our model can synthesize high-resolution key-frames using the low-resolution data stream. Finally, Fast-Vid2Vid++ interpolates intermediate frames by motion compensation with slight latency and generates full-length sequences with motion-aware inference (MAI). On standard benchmarks, Fast-Vid2Vid++ achieves a real-time performance of 30-59 FPS and saves 28-35× computational cost on a single V100 GPU. Code and models are publicly available.

Deep Learning-based Perceptual Video Quality Enhancement for 3D Synthesized View

Convolutional Neural Network Based Synthesized View Quality Enhancement for 3D Video Coding

High-Efficiency 3D Depth Coding Based on Perceptual Quality of Synthesized Video.

No-Reference Multi-Level Video Quality Assessment Metric for 3D-Synthesized Videos

Synthesis-Aware Region-Based 3D Video Coding.

Efficient Depth Map Compression For View Rendering In 3d Video

Compression-Realized Deep Structural Network for Video Quality Enhancement

A Generative Adversarial Network to Denoise Depth Maps for Quality Improvement of DIBR-Synthesized Stereoscopic Images

Deep Learning Based Just Noticeable Difference and Perceptual Quality Prediction Models for Compressed Video

Temporal Denoising Mask Synthesis Network for Learning Blind Video Temporal Consistency

Depth Estimation of Video Sequences with Perceptual Losses

Saliency-Based Joint Distortion Model For 3d Video Coding

Synthesis distortion estimation in 3D video using frequency and spatial analysis

Video Deblurring Via Spatiotemporal Pyramid Network and Adversarial Gradient Prior

Stereoscopic Video Quality Assessment Based on 3D Convolutional Neural Networks

Depth No-Synthesis-Error Model for View Synthesis in 3-D Video

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

Learning-Based Stereoscopic View Synthesis with Cascaded Deep Neural Networks

Fast-Vid2Vid++: Spatial-Temporal Distillation for Real-Time Video-to-Video Synthesis

Deep Video‐Based Performance Synthesis from Sparse Multi‐View Capture

Deep Convolutional Neural Network For Decompressed Video Enhancement