Abstract:Recent advancements in large-scale video-language models have shown significant potential for real-time planning and detailed interactions. However, their high computational demands and the scarcity of annotated datasets limit their practicality for academic researchers. In this work, we introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens within bridge layers to allow for the encoding of entire video sequences alongside historical visual data, effectively preserving semantic continuity and enhancing model performance across various tasks. This approach includes recurrent memory tokens and a SceneTilling algorithm, which segments videos into independent semantic units to preserve semantic integrity. Empirically, VideoLLaMB significantly outstrips existing video-language models, demonstrating a 5.5 points improvement over its competitors across three VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive results on the MVBench show that VideoLLaMB-7B achieves markedly better results than previous 7B models of same LLM. Remarkably, it maintains robust performance as PLLaVA even as video length increases up to 8 times. Besides, the frame retrieval results on our specialized Needle in a Video Haystack (NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately identifying specific frames within lengthy videos. Our SceneTilling algorithm also enables the generation of streaming video captions directly, without necessitating additional training. In terms of efficiency, VideoLLaMB, trained on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear GPU memory scaling, ensuring both high performance and cost-effectiveness, thereby setting a new foundation for long-form video-language models in both academic and practical applications.

Jointly Modeling Embedding and Translation to Bridge Video and Language

Video Captioning with Transferred Semantic Attributes.

CC-LSTM: Cross and Conditional Long-Short Time Memory for Video Captioning

Video Captioning With Attention-Based LSTM and Semantic Consistency

Bidirectional Long-Short Term Memory for Video Description

Attention-based LSTM with Semantic Consistency for Videos Captioning

Describing Video with Attention-Based Bidirectional LSTM

TEVL: Trilinear Encoder for Video-language Representation Learning

Visual Oriented Encoder: Integrating Multimodal and Multi-Scale Contexts for Video Captioning

Exploiting long-term temporal dynamics for video captioning

Learning Language-Visual Embedding for Movie Understanding with Natural-Language

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Text2Video: an End-to-end Learning Framework for Expressing Text with Videos

Hierarchical Multimodal Lstm For Dense Visual-Semantic Embedding

From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning

VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges

Emotional Video Captioning With Vision-Based Emotion Interpretation Network

Bridge Video and Text with Cascade Syntactic Structure.

A novel Multi-Layer Attention Framework for visual description prediction using bidirectional LSTM

Dual-Stream Recurrent Neural Network for Video Captioning

Extraction method of semantic information of video images