Abstract:Modern codecs remove temporal redundancy of a video via inter prediction, i.e., searching previously coded frames for similar blocks and storing motion vectors to save bit-rates. However, existing codecs adopt blocklevel motion estimation, where a block is regressed by reference blocks linearly and is doomed to fail to deal with non-linear motions. In this article, we generate virtual reference frames (VRFs) with previously reconstructed frames via deep networks to offer an additional candidate, which is not constrained to linear motion structure and further significantly improves coding efficiency. More specifically, we propose a novel deep Auto-Regressive Moving-Average (ARMA) model, Error-Corrected Auto-Regressive Network (ECARNet), equipped with the powers of the conventional statistic ARMA models and deep networks jointly for reference frame prediction. Similar to conventionalARMAmodels, the ECAR-Net consists of two stages: AutoRegression (AR) stage and Error-Correction (EC) stage, where the first part predicts the signal at the current time-step based on previously reconstructed frames, while the second one compensates for the output of the AR stage to obtain finer details. Different from the statistic AR models only focusing on short-term temporal dependency, the AR model of our ECAR-Net is further injected with the long-term dynamics mechanism, where long temporal information is utilized to help predict motions more accurately. Furthermore, ECAR-Net works in a configuration-adaptive way, i.e., using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios. With the well-designed network, our method surpasses HEVC on average 5.0% and 6.6% BD-rate saving for the luma component under the Low Delay B and Random Access configurations and also obtains on average 1.54% BD-rate saving over VVC. Furthermore, ECAR-Net works in a configuration-adaptive way, i.e., using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios.

Advanced residual prediction enhancement for 3D-HEVC

Advanced Residual Prediction In 3d-Hevc

Low-complexity advanced residual prediction design in 3D-HEVC

A 3D-HEVC Fast Mode Decision Algorithm for Real-Time Applications

Disparity Vector Based Advanced Inter-View Prediction in 3D-HEVC.

Linear adaptive search range model for uni-prediction and motion analysis for bi-prediction in HEVC

Texture and Correlation Based Fast Intra Prediction Algorithm for HEVC

Enhanced Ctu-Level Inter Prediction with Deep Frame Rate Up-Conversion for High Efficiency Video Coding

Inter-view motion vector prediction for depth coding

Neural Network-Based Enhancement to Inter Prediction for Video Coding

Improved Disparity Vector Derivation in 3D-HEVC

Disparity-compensated Inter-Layer Motion Prediction Using Standardized HEVC Extensions.

Efficient Prediction Methods With Enhanced Spatial-Temporal Correlation for HEVC.

Deep Inter Prediction with Error-Corrected Auto-Regressive Network for Video Coding.

Neural Network Based Inter Prediction for HEVC

Efficient Multiple-Line-Based Intra Prediction for HEVC.

Convolutional Neural Network-Based Coefficients Prediction for HEVC Intra-Predicted Residues

Deep Multi-Domain Prediction for 3D Video Coding.

Enhanced Motion-Compensated Video Coding with Deep Virtual Reference Frame Generation

Low complexity Neighboring Block based Disparity Vector Derivation in 3D-HEVC

Memory-Augmented Auto-Regressive Network for Frame Recurrent Inter Prediction.