Abstract:As a rapid development of neural-network-based machine learning algorithms, deep learning methods are being tentatively used in a much wider range than well-known artificial intelligence applications such as face recognition or auto-driving. Recently, deep learning models are investigated intensively to improve the compression efficiency for video coding, especially at the in-loop filtering stage. Although deep learning-based in-loop filtering methods in prior arts have already shown a remarkable potential capability in video coding, content propagation issue is still not well recognized and addressed yet. Content propagation is the fact that contents of reference frames are propagated to frames referring to them, which typically leads to over-filtering issues. In this article, we develop an iteratively trained deep in-loop filter with adaptive model selection (iDAM) to address the content propagation issue. First, we propose an iterative training scheme, which enables the network to gradually take into account the impacts of content propagation. Second, we propose a filter selection mechanism, i.e., allowing a block to select from a set of candidate filters with different filtering strengths. Besides, we propose a novel approach to design a conditional in-loop filtering method that can deal with multiple quality levels with a single model and serve the functionality of filter selection by modifying the input parameters. Extensive experiments on top of the latest video coding standard (Versatile Video Coding, VVC) have been conducted to evaluate the proposed techniques. Compared with VTM-11.0, our scheme achieves a new state-of-the-art, leading to {7.91%, 20.25%, 20.44%}, {11.64%, 26.40%, 26.50%}, and {10.97%, 26.63%, 26.77%} BD-rate reductions on average for {Y, Cb, Cr} under all-intra, random-access, and low-delay configurations, respectively. As far as we know, our proposed iDAM scheme provides the highest coding performance compared to all existing solutions. In addition, the syntax elements of the proposed scheme were adopted at the 76th meeting of Audio Video coding Standard (AVS) held this year.

An Attention Based CNN with Temporal Hierarchical Deployment for AVS3 Inter In-loop Filtering

Residual in Residual Based Convolutional Neural Network In-loop Filter for AVS3

CNN-Based Inter Prediction Refinement for AVS3

NR-CNN: Nested-Residual Guided CNN In-loop Filtering for Video Coding

Content-Aware Convolutional Neural Network for In-Loop Filtering in High Efficiency Video Coding

An Integrated CNN-based Post Processing Filter For Intra Frame in Versatile Video Coding

Multi-Type Self-Attention-Based Convolutional-Neural-Network Post-Filtering for AV1 Codec

Towards Next Generation Video Coding: from Neural Network Based Predictive Coding to In-Loop Filtering

A CU-level Adaptive Decision Method for CNN-based In-Loop Filtering

Multi-Density Convolutional Neural Network for In-Loop Filter in Video Coding.

Convolutional Neural Network Based In-Loop Filter for VVC Intra Coding

Multi-Gradient Convolutional Neural Network Based In-Loop Filter For Vvc

Adaptive CU Split Decision with Pooling-variable CNN for VVC Intra Encoding.

Multi-Density Attention Network for Loop Filtering in Video Compression

A CNN-Based In-Loop Filter with CU Classification for HEVC

One-for-All: an Efficient Variable Convolution Neural Network for In-Loop Filter of VVC

Gated fusion network for SAO filter and inter frame prediction in Versatile Video Coding

Joint Luma and Chroma Multi-Scale CNN In-loop Filter for Versatile Video Coding

Idam: Iteratively Trained Deep In-Loop Filter with Adaptive Model Selection

Lightweight Multiattention Recursive Residual CNN-based In-loop Filter driven by Neuron Diversity

Intra Block Partition Structure Prediction Via Convolutional Neural Network.