Abstract:Multimodal Large Language Models (MLLMs) are widely used for visual perception, understanding, and reasoning. However, long video processing and precise moment retrieval remain challenging due to LLMs' limited context size and coarse frame extraction. We propose the Large Language-and-Vision Assistant for Moment Retrieval (LLaVA-MR), which enables accurate moment retrieval and contextual grounding in videos using MLLMs. LLaVA-MR combines Dense Frame and Time Encoding (DFTE) for spatial-temporal feature extraction, Informative Frame Selection (IFS) for capturing brief visual and motion patterns, and Dynamic Token Compression (DTC) to manage LLM context limitations. Evaluations on benchmarks like Charades-STA and QVHighlights demonstrate that LLaVA-MR outperforms 11 state-of-the-art methods, achieving an improvement of 1.82% in R1@0.5 and 1.29% in mAP@0.5 on the QVHighlights dataset. Our implementation will be open-sourced upon acceptance.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is: **the challenges faced by multimodal large - language models (MLLMs) in performing precise moment retrieval (Video Moment Retrieval, MR) in long - videos**. Specifically, these challenges include: 1. **Insufficient time - awareness and understanding**: When dealing with long - time videos, existing MLLMs usually rely on sparse sampling due to their limited context length, which makes it difficult for them to capture short - but - important visual details, thus affecting the precise understanding of video content and time - location. 2. **Difficulty in precisely locating key moments**: Precise location of short, important moments in long - videos is a difficult problem. Due to the limited context size of MLLMs, they often rely on sparse sampling, which reduces their ability to capture instantaneous visual changes. To solve these problems, the author proposes **Large - Language and Visual Assistant for Moment Retrieval (LLaVA - MR)**, which enhances the time - awareness ability and key - moment - capturing ability of MLLMs through the following technical means: - **Dense Frame and Time Encoding (DFTE)**: Used to extract fine - grained spatial and temporal features and improve time - awareness precision. - **Informative Frame Selection (IFS)**: Used to capture short - but - important visual and motion patterns. - **Dynamic Token Compression (DTC)**: Used to reduce the sequence length while retaining important information, enabling MLLMs to more effectively process long - time video - frame sequences. Through these improvements, LLaVA - MR performs well in multiple benchmark tests, especially when dealing with complex, long - time videos, achieving significant performance improvements. For example, on the QVHighlights dataset, LLaVA - MR has increased by 1.82% and 1.29% respectively in the R1@0.5 and mAP@0.5 metrics.

LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval

Context-Enhanced Video Moment Retrieval with Large Language Models

TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations

The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval

Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification

VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM

PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

PG-Video-LLaVA: Pixel Grounding Large Video-Language Models

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models

LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval

Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs