Abstract:Recent years have witnessed rapid advances in learnt video coding. Most algorithms have solely relied on the vector-based motion representation and resampling (e.g., optical flow based bilinear sampling) for exploiting the inter frame redundancy. In spite of the great success of adaptive kernel-based resampling (e.g., adaptive convolutions and deformable convolutions) in video prediction for uncompressed videos, integrating such approaches with rate-distortion optimization for inter frame coding has been less successful. Recognizing that each resampling solution offers unique advantages in regions with different motion and texture characteristics, we propose a hybrid motion compensation (HMC) method that adaptively combines the predictions generated by these two approaches. Specifically, we generate a compound spatiotemporal representation (CSTR) through a recurrent information aggregation (RIA) module using information from the current and multiple past frames. We further design a one-to-many decoder pipeline to generate multiple predictions from the CSTR, including vector-based resampling, adaptive kernel-based resampling, compensation mode selection maps and texture enhancements, and combines them adaptively to achieve more accurate inter prediction. Experiments show that our proposed inter coding system can provide better motion-compensated prediction and is more robust to occlusions and complex motions. Together with jointly trained intra coder and residual coder, the overall learnt hybrid coder yields the state-of-the-art coding efficiency in low-delay scenario, compared to the traditional H.264/AVC and H.265/HEVC, as well as recently published learning-based methods, in terms of both PSNR and MS-SSIM metrics.

End to End Alignment Learning of Instructional Videos with Spatiotemporal Hybrid Encoding and Decoding Space Reduction

End-to-end Neural Video Coding Using a Compound Spatiotemporal Representation

Alignment-guided Temporal Attention for Video Action Recognition

Aligning Videos in Space and Time

Collaborative Spatio-temporal Feature Learning for Video Action Recognition

WINNER: Weakly-Supervised Hierarchical Decomposition and Alignment for Spatio-tEmporal Video Grounding

Learning Semantic Concepts and Temporal Alignment for Narrated Video Procedural Captioning

Optimized Embedded Model for Time Sequence Alignment Within Distributed Video Streams Environment

Video alignment using unsupervised learning of local and global features

Learning by Aligning 2D Skeleton Sequences and Multi-Modality Fusion

Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network

Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions.

Implicit Temporal Modeling with Learnable Alignment for Video Recognition

Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment

Context-Aware Sequence Alignment using 4D Skeletal Augmentation

AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

End-to-End Spatio-Temporal Action Localisation with Video Transformers

Video-Language Alignment via Spatio-Temporal Graph Transformer

Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding

Align and Tell: Boosting Text-Video Retrieval With Local Alignment and Fine-Grained Supervision

You Only Align Once: Bidirectional Interaction for Spatial-Temporal Video Super-Resolution