PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

Ruyang Liu,Haoran Tang,Haibo Liu,Yixiao Ge,Ying Shan,Chen Li,Jiankun Yang
2024-11-05
Abstract:The past year has witnessed the significant advancement of video-based large language models. However, the challenge of developing a unified model for both short and long video understanding remains unresolved. Most existing video LLMs cannot handle hour-long videos, while methods custom for long videos tend to be ineffective for shorter videos and images. In this paper, we identify the key issue as the redundant content in videos. To address this, we propose a novel pooling strategy that simultaneously achieves token compression and instruction-aware visual feature aggregation. Our model is termed Prompt-guided Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three core components: the CLIP-based visual-prompt alignment that extracts visual information relevant to the user's instructions, the prompt-guided pooling that compresses the visual sequence to arbitrary scales using convolution-style pooling, and the clip context extension designed for lengthy prompt common in visual dialogue. Moreover, our codebase also integrates the most advanced video Direct Preference Optimization (DPO) and visual interleave training. Extensive experiments have validated the performance of our model. With superior throughput and only 1024 visual context, PPLLaVA achieves better results on image benchmarks as a video LLM, while achieving state-of-the-art performance across various video benchmarks, excelling in tasks ranging from caption generation to multiple-choice questions, and handling video lengths from seconds to hours. Codes have been available at <a class="link-external link-https" href="https://github.com/farewellthree/PPLLaVA" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The main problem this paper attempts to address is the development of a unified model capable of handling both short and long videos. Most existing Video Large Language Models (Video LLMs) perform poorly when processing long videos, while methods specifically designed for long videos are limited in their effectiveness when dealing with short videos and images. The paper points out that redundant content in videos is the key issue. To tackle this challenge, the authors propose a new pooling strategy called Prompt-guided Pooling, which can compress visual sequences while achieving instruction-aware visual feature aggregation. Specifically, the paper proposes a model named Prompt-guided Pooling LLaVA (PPLLaVA), which includes three core components: 1. **CLIP-based Visual Prompt Alignment**: Extracts visual information relevant to user instructions. 2. **Prompt-guided Pooling**: Uses a convolution-style pooling method to compress visual sequences to any scale. 3. **Segment Context Extension**: Designed to handle long prompts common in visual dialogues. Through these innovations, PPLLaVA can efficiently handle videos of different lengths and achieves state-of-the-art performance on multiple video benchmarks, covering tasks ranging from a few seconds to several hours. Additionally, the model performs well on image benchmarks and has higher throughput.