Abstract:Multi-modal large language models (MLLMs) have demonstrated considerable potential across various downstream tasks that require cross-domain knowledge. MLLMs capable of processing videos, known as Video-MLLMs, have attracted broad interest in video-language understanding. However, videos, especially long videos, contain more visual tokens than images, making them difficult for LLMs to process. Existing works either downsample visual features or extend the LLM context size, risking the loss of high-resolution information or slowing down inference speed. To address these limitations, we apply cross-attention layers in the intermediate projector between the visual encoder and the large language model (LLM). As the naive cross-attention mechanism is insensitive to temporal order, we further introduce causal cross-attention masks (CCAMs) within the cross-attention layers. This Video-MLLM, named Video-CCAM, is trained in a straightforward two-stage fashion: feature alignment and visual instruction tuning. We develop several Video-CCAM models based on LLMs of different sizes (4B, 9B, and 14B). Video-CCAM proves to be a robust Video-MLLM and shows outstanding performance from short videos to long ones. Among standard video benchmarks like MVBench and VideoChatGPT-QA, Video-CCAM shows outstanding performances (1st/2nd/3rd in MVBench and TGIF-QA, 2nd/3rd/4th in MSVD-QA, MSRVTT-QA, and ActivityNet-QA). In benchmarks encompassing long videos, Video-CCAM models can be directly adapted to long video understanding and still achieve exceptional scores despite being trained solely with images and 16-frame videos. Using 96 frames (6$\times$ the training number of frames), Video-CCAM models rank 1st/2nd/3rd in VideoVista and 1st/2nd/4th in MLVU among all open-source Video-MLLMs, respectively. The code is publicly available in \url{<a class="link-external link-https" href="https://github.com/QQ-MM/Video-CCAM" rel="external noopener nofollow">this https URL</a>}.

Multi-layer multi-view topic model for classifying advertising video.

Multi-View Analysis Dictionary Learning for Image Classification.

An Unsupervised Video Summarization Method Based on Multimodal Representation.

Multi-label Learning with Label Relevance in Advertising Video

Multi-modal Representation Learning for Video Advertisement Content Structuring

Videoader: a video advertising system based on intelligent analysis of visual content

A Multimodal Framework for Video Ads Understanding

Predicting Content Similarity Via Multimodal Modeling for Video-In-Video Advertising.

TV Ad Video Categorization with Probabilistic Latent Concept Learning

Multimodal Deep Representation Learning for Video Classification

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

MM-AU:Towards Multimodal Understanding of Advertisement Videos

Mining Adjacent Markets from a Large-Scale Ads Video Collection for Image Advertising

Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification.

Learning Dual Low-Rank Representation for Multi-Label Micro-Video Classification.

Multimodal Fusion and Coherence Modeling for Video Topic Segmentation

Dual-domain Aligned Deep Hierarchical Matrix Factorization Method for Micro-video Multi-label Classification

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

Towards Good Practices for Multi-modal Fusion in Large-scale Video Classification

MTAF: Shopping Guide Micro-Videos Popularity Prediction Using Multimodal and Temporal Attention Fusion Approach.

A Novel Fusion Method for Semantic Concept Classification in Video