Abstract:Over the past two decades, traditional block-based video coding has made remarkable progress and spawned a series of well-known standards such as MPEG-4, H.264/AVC and H.265/HEVC. On the other hand, deep neural networks (DNNs) have shown their powerful capacity for visual content understanding, feature extraction and compact representation. Some previous works have explored the learnt video coding algorithms in an end-to-end manner, which show the great potential compared with traditional methods. In this paper, we propose an end-to-end deep neural video coding framework (NVC), which uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals, respectively. Novel features of NVC include: 1) To estimate and compensate motion over a large range of magnitudes, we propose an unsupervised multiscale motion compensation network (MS-MCN) together with a pyramid decoder in the VAE for coding motion features that generates multiscale flow fields, 2) we design a novel adaptive spatiotemporal context model for efficient entropy coding for motion information, 3) we adopt nonlocal attention modules (NLAM) at the bottlenecks of the VAEs for implicit adaptive feature extraction and activation, leveraging its high transformation capacity and unequal weighting with joint global and local information, and 4) we introduce multi-module optimization and a multi-frame training strategy to minimize the temporal error propagation among P-frames. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods following the common test conditions, demonstrating consistent gains across all popular test sequences for both PSNR and MS-SSIM distortion metrics.

Deep Neural Network Based Frame Reconstruction for Optimized Video Coding

Towards Coding for VoD Application: an Enhanced Video Compression System with a Content-Fitted Recursive Restoration Network

Advances in Video Compression System Using Deep Neural Network: A Review and Case Studies

Improved Low-Bitrate HEVC Video Coding Using Deep Learning Based Super-Resolution and Adaptive Block Patching.

A Slimmable Framework for Practical Neural Video Compression

Deep Convolutional Neural Network For Decompressed Video Enhancement

DeepCoder: A Deep Neural Network Based Video Compression

SR-CL-DMC: P-frame coding with Super-Resolution, Color Learning, and Deep Motion Compensation

High Efficiency Deep-learning Based Video Compression

Deep Frame Prediction for Video Coding

Rate-Performance-Loss Optimization for Inter-Frame Deep Feature Coding from Videos

DVC: An End-to-end Deep Video Compression Framework

Down-Sampling Based Video Coding with Degradation-Aware Restoration-Reconstruction Deep Neural Network

High-Efficiency Neural Video Compression via Hierarchical Predictive Learning

Towards Lightweight Deep Reference Frame for Versatile Video Coding

Multi-rate Adaptive Transform Coding for Video Compression

A Neural-network Enhanced Video Coding Framework beyond ECM

A Coding Framework and Benchmark towards Low-Bitrate Video Understanding

Neural Video Coding Using Multiscale Motion Compensation and Spatiotemporal Context Model

Deep Learning-Based Video Coding

Preprocessing for Multi-Dimensional Enhancement and Reconstruction in Neural Video Compression