Abstract:Video compression is indispensable to most video analysis systems. Despite saving the transportation bandwidth, it also deteriorates downstream video understanding tasks, especially at low-bitrate settings. To systematically investigate this problem, we first thoroughly review the previous methods, revealing that three principles, i.e., task-decoupled, label-free, and data-emerged semantic prior, are critical to a machine-friendly coding framework but are not fully satisfied so far. In this paper, we propose a traditional-neural mixed coding framework that simultaneously fulfills all these principles, by taking advantage of both traditional codecs and neural networks (NNs). On one hand, the traditional codecs can efficiently encode the pixel signal of videos but may distort the semantic information. On the other hand, highly non-linear NNs are proficient in condensing video semantics into a compact representation. The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w.r.t. the coding procedure, which is spontaneously learned from unlabeled data in a self-supervised manner. The videos collaboratively decoded from two streams (codec and NN) are of rich semantics, as well as visually photo-realistic, empirically boosting several mainstream downstream video analysis task performances without any post-adaptation procedure. Furthermore, by introducing the attention mechanism and adaptive modeling scheme, the video semantic modeling ability of our approach is further enhanced. Fianlly, we build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach. All codes, data, and models will be open-sourced for facilitating future research.

Nearest neighbor search with compact codes: A decoder perspective

Polysemous codes

Fast Nearest-Neighbor Codeword Search Algorithms for Vector Quantization

Fast Norm-Ordered Nearest-Neighbor Codeword Search Algorithms For Image Vector Quantization

Efficient approximate nearest neighbor search with integrated binary codes.

Learning Compact Binary Codes Via Pairwise Correlation Reconstruction

Derived Codebooks for High-Accuracy Nearest Neighbor Search

Deep Visual-Semantic Quantization For Efficient Image Retrieval

Fast codevector search scheme for 3D mesh model vector quantisation

Equal-Average Equal-Variance Nearest Neighbor Search Algorithm Based on Hadamard Transform

Fast Nearest Neighbor Search in the Hamming Space.

SUBIC: A supervised, structured binary code for image search

Ordinal Constraint Binary Coding for Approximate Nearest Neighbor Search

Efficient Neural Compression with Inference-time Decoding

A Compact Binary Aggregated Descriptor Via Dual Selection for Visual Search

Equal-Average Equal-Variance Equal-Norm Nearest Neighbor Search Algorithm for Vector Quantization

Quadra-embedding: Binary code embedding with low quantization error

Double-Bit Quantization and Index Hashing for Nearest Neighbor Search

A Coding Framework and Benchmark towards Low-Bitrate Video Understanding

Accelerated Nearest Neighbor Search with Quick ADC

Neural Distributed Compressor Discovers Binning