Abstract:Multi-modal large language models (MLLMs) have demonstrated considerable potential across various downstream tasks that require cross-domain knowledge. MLLMs capable of processing videos, known as Video-MLLMs, have attracted broad interest in video-language understanding. However, videos, especially long videos, contain more visual tokens than images, making them difficult for LLMs to process. Existing works either downsample visual features or extend the LLM context size, risking the loss of high-resolution information or slowing down inference speed. To address these limitations, we apply cross-attention layers in the intermediate projector between the visual encoder and the large language model (LLM). As the naive cross-attention mechanism is insensitive to temporal order, we further introduce causal cross-attention masks (CCAMs) within the cross-attention layers. This Video-MLLM, named Video-CCAM, is trained in a straightforward two-stage fashion: feature alignment and visual instruction tuning. We develop several Video-CCAM models based on LLMs of different sizes (4B, 9B, and 14B). Video-CCAM proves to be a robust Video-MLLM and shows outstanding performance from short videos to long ones. Among standard video benchmarks like MVBench and VideoChatGPT-QA, Video-CCAM shows outstanding performances (1st/2nd/3rd in MVBench and TGIF-QA, 2nd/3rd/4th in MSVD-QA, MSRVTT-QA, and ActivityNet-QA). In benchmarks encompassing long videos, Video-CCAM models can be directly adapted to long video understanding and still achieve exceptional scores despite being trained solely with images and 16-frame videos. Using 96 frames (6$\times$ the training number of frames), Video-CCAM models rank 1st/2nd/3rd in VideoVista and 1st/2nd/4th in MLVU among all open-source Video-MLLMs, respectively. The code is publicly available in \url{<a class="link-external link-https" href="https://github.com/QQ-MM/Video-CCAM" rel="external noopener nofollow">this https URL</a>}.

RepCaM: Re-parameterization Content-aware Modulation for Neural Video Delivery.

Overfitting the Data: Compact Neural Video Delivery Via Content-aware Feature Modulation

Efficient Meta-Tuning for Content-Aware Neural Video Delivery.

Video super-resolution with phase-aided deformable alignment network

Deep-Reinforcement-Learning-based User-Preference-Aware Rate Adaptation for Video Streaming

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Releasing the Parameter Latency of Neural Representation for High-Efficiency Video Compression

Video Compression with Arbitrary Rescaling Network

Cascaded Temporal Updating Network for Efficient Video Super-Resolution

Rate-aware Compression for NeRF-based Volumetric Video

A Coding Framework and Benchmark towards Low-Bitrate Video Understanding

RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization

Rate-Quality Based Rate Control Model for Neural Video Compression.

HPC: Hierarchical Progressive Coding Framework for Volumetric Video

Collaborative Feedback Discriminative Propagation for Video Super-Resolution

Content-Adaptive Rate Control Method for User-Generated Content Videos

MetaSCI: Scalable and Adaptive Reconstruction for Video Compressive Sensing

RepNeXt: A Fast Multi-Scale CNN using Structural Reparameterization

Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression

Content-adaptive Variable Resolution Framework for Intra Coding

Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing