Abstract:Compressed videos offer a compelling alternative to raw videos, showing the possibility to significantly reduce the on-line computational and storage cost. However, current approaches to compressed video processing generally follow the resource-consuming pre-training and fine-tuning paradigm, which does not fully take advantage of such properties, making them not favorable enough for widespread applications. Inspired by recent successes of prompt tuning techniques in computer vision, this paper presents the first attempt to build a prompt based representation learning framework, which enables effective and efficient adaptation of pre-trained raw video models to compressed video understanding tasks. To this end, we propose a novel prompt tuning approach, namely Compressed Video Prompt Tuning (CVPT), emphatically dealing with the challenging issue caused by the inconsistency between pre-training and downstream data modalities. Specifically, CVPT replaces the learnable prompts with compressed modalities (\emph{e.g.} Motion Vectors and Residuals) by re-parameterizing them into conditional prompts followed by layer-wise refinement. The conditional prompts exhibit improved adaptability and generalizability to instances compared to conventional individual learnable ones, and the Residual prompts enhance the noisy motion cues in the Motion Vector prompts for further fusion with the visual cues from I-frames. Additionally, we design Selective Cross-modal Complementary Prompt (SCCP) blocks. After inserting them into the backbone, SCCP blocks leverage semantic relations across diverse levels and modalities to improve cross-modal interactions between prompts and input flows. Extensive evaluations on HMDB-51, UCF-101 and Something-Something v2 demonstrate that CVPT remarkably outperforms the state-of-the-art counterparts, delivering a much better balance between accuracy and efficiency.

TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception

Prompt-ICM: A Unified Framework towards Image Coding for Machines with Task-driven Prompts

Compressed Video Prompt Tuning.

Transformer-based Variable-rate Image Compression with Region-of-interest Control

Transformer-based Image Compression with Variable Image Quality Objectives

TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding

From Patch to Pixel: A Transformer-Based Hierarchical Framework for Compressive Image Sensing.

Progressive Learning with Visual Prompt Tuning for Variable-Rate Image Compression

Prompt-Free Diffusion: Taking "text" out of Text-to-Image Diffusion Models

Learned Image Coding for Human-Machine Collaborative Optimization

AICT: An Adaptive Image Compression Transformer

Task-Driven Video Compression for Humans and Machines: Framework Design and Optimization

Exploring Compressed Image Representation as a Perceptual Proxy: A Study

A Unified Image Compression Method for Human Perception and Multiple Vision Tasks

Composable Image Coding for Machine Via Task-oriented Internal Adaptor and External Prior

Towards On-demand Transmission: Joint Feature and Image Coding with Reversible Neural Networks

Transformer-based Learned Image Compression for Joint Decoding and Denoising

Preprocessing Enhanced Image Compression for Machine Vision

Dynamic and Compressive Adaptation of Transformers From Images to Videos

A Preprocessing Framework for Video Machine Vision under Compression

Towards End-to-End Image Compression and Analysis with Transformers