Abstract:In modern video coding standards, block-based inter prediction is widely adopted, which brings high compression efficiency. However, in natural videos, there are usually multiple moving objects of arbitrary shapes, resulting in complex motion fields that are difficult to represent compactly. This problem has been tackled by more flexible block partitioning methods in the Versatile Video Coding (VVC) standard, but the more flexible partitions require more overhead bits to signal and still cannot be made arbitrarily shaped. To address this limitation, we propose an object segmentation-assisted inter prediction method (SAIP), where objects in the reference frames are segmented by some advanced technologies. With a proper indication, the object segmentation mask is translated from the reference frame to the current frame as the arbitrary-shaped partition of different regions without any extra signal. Using the segmentation mask, motion compensation is separately performed for different regions, achieving higher prediction accuracy. The segmentation mask is further used to code the motion vectors of different regions more efficiently. Moreover, the segmentation mask is considered in the joint rate-distortion optimization for motion estimation and partition estimation to derive the motion vector of different regions and partition more accurately. The proposed method is implemented into the VVC reference software, VTM version 12.0. Experimental results show that the proposed method achieves up to 1.98%, 1.14%, 0.79%, and on average 0.82%, 0.49%, 0.37% BD-rate reduction for common test sequences, under the Low-delay P, Low-delay B, and Random Access configurations, respectively.

Block-correlation-based Intra Prediction for VVC

Intra Prediction and Mode Coding in VVC

Fusion-Based Versatile Video Coding Intra Prediction Algorithm with Template Matching and Linear Prediction

Interweaved Prediction for Video Coding.

Motion Vector Coding and Block Merging in the Versatile Video Coding Standard

Adaptive Chroma Block Vector Derivation from Luma for Screen Content Coding

Towards Lightweight Neural Network-based Chroma Intra Prediction for Video Coding

Joint Local and Nonlocal Progressive Prediction for Versatile Video Coding

Efficient Cross-Component Prediction for H.266/VVC Based on Lightweight Fully Connected Networks

Texture and Correlation Based Fast Intra Prediction Algorithm for HEVC

Unified Intra Mode Coding Based on Short and Long Range Correlations.

Spatio-Temporal Convolutional Neural Network for Enhanced Inter Prediction in Video Coding

Learning-Based Fast Splitting and Directional Mode Decision for VVC Intra Prediction

Neural network-based cross-channel chroma prediction for versatile video coding

Spatial Information Refinement for Chroma Intra Prediction in Video Coding

Multi-view Video Coding Based on View Prediction

Bi-Intra Prediction for Versatile Video Coding

Chroma Intra Prediction with Lightweight Attention-Based Neural Networks

Faster Intra-Prediction of Versatile Video Coding Using a Concatenate-Designed CNN via DCT Coefficients

Object Segmentation-Assisted Inter Prediction for Versatile Video Coding

High-Order Intra Prediction for Future Video Coding