Abstract:Video compression is indispensable to most video analysis systems. Despite saving the transportation bandwidth, it also deteriorates downstream video understanding tasks, especially at low-bitrate settings. To systematically investigate this problem, we first thoroughly review the previous methods, revealing that three principles, i.e., task-decoupled, label-free, and data-emerged semantic prior, are critical to a machine-friendly coding framework but are not fully satisfied so far. In this paper, we propose a traditional-neural mixed coding framework that simultaneously fulfills all these principles, by taking advantage of both traditional codecs and neural networks (NNs). On one hand, the traditional codecs can efficiently encode the pixel signal of videos but may distort the semantic information. On the other hand, highly non-linear NNs are proficient in condensing video semantics into a compact representation. The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w.r.t. the coding procedure, which is spontaneously learned from unlabeled data in a self-supervised manner. The videos collaboratively decoded from two streams (codec and NN) are of rich semantics, as well as visually photo-realistic, empirically boosting several mainstream downstream video analysis task performances without any post-adaptation procedure. Furthermore, by introducing the attention mechanism and adaptive modeling scheme, the video semantic modeling ability of our approach is further enhanced. Fianlly, we build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach. All codes, data, and models will be open-sourced for facilitating future research.

Semantic classifier based on compressed sensing for image and video annotation

Image Classification Method by Combining Multi-features and Sparse Coding

Semantic Text Compression for Classification

Semantic-Aware Image Compressed Sensing

Adjacent Coding for Image Classification.

Compressed Domain Motion Analysis for Video Semantic Events Detection

Video News Indexing Using Semantic-Face

Image annotation by semantic sparse recoding of visual content.

Semantic Sparse Recoding of Visual Content for Image Applications

Robust Semantic Video Indexing by Harvesting Web Images.

Semantic similarity information discrimination for video captioning

Compression Ratio Learning and Semantic Communications for Video Imaging

SSSIC: Semantics-to-Signal Scalable Image Coding with Learned Structural Representations.

A video compression-cum-classification network for classification from compressed video streams

Leveraging Compressed Frame Sizes For Ultra-Fast Video Classification

Applying Semantic Association To Support Content-Based Video Retrieval

A Novel Image/video Coding Method Based on Compressed Sensing Theory

Semantics-to-Signal Scalable Image Compression with Learned Revertible Representations

Video diver: generic video indexing with diverse features.

An Automatic Video Annotation Method Based on Multiple Complementary Classifiers

A Coding Framework and Benchmark towards Low-Bitrate Video Understanding