Abstract:Recently, Vision Large Language Models (VLLMs) integrated with vision encoders have shown promising performance in vision understanding. The key of VLLMs is to encode visual content into sequences of visual tokens, enabling VLLMs to simultaneously process both visual and textual content. However, understanding videos, especially long videos, remain a challenge to VLLMs as the number of visual tokens grows rapidly when encoding videos, resulting in the risk of exceeding the context window of VLLMs and introducing heavy computation burden. To restrict the number of visual tokens, existing VLLMs either: (1) uniformly downsample videos into a fixed number of frames or (2) reducing the number of visual tokens encoded from each frame. We argue the former solution neglects the rich temporal cue in videos and the later overlooks the spatial details in each frame. In this work, we present Balanced-VLLM (B-VLLM): a novel VLLM framework that aims to effectively leverage task relevant spatio-temporal cues while restricting the number of visual tokens under the VLLM context window length. At the core of our method, we devise a text-conditioned adaptive frame selection module to identify frames relevant to the visual understanding task. The selected frames are then de-duplicated using a temporal frame token merging technique. The visual tokens of the selected frames are processed through a spatial token sampling module and an optional spatial token merging strategy to achieve precise control over the token count. Experimental results show that B-VLLM is effective in balancing the number of frames and visual tokens in video understanding, yielding superior performance on various video understanding benchmarks. Our code is available at <a class="link-external link-https" href="https://github.com/zhuqiangLu/B-VLLM" rel="external noopener nofollow">this https URL</a>.

LVLM-COUNT: Enhancing the Counting Ability of Large Vision-Language Models

Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning

Overcoming Statistical Shortcuts for Open-ended Visual Counting

VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance

VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models

Q-VLM: Post-training Quantization for Large Vision-Language Models

B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

CountCLIP -- [Re] Teaching CLIP to Count to Ten

Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models

Counting Ability of Large Language Models and Impact of Tokenization

Improved Baselines with Visual Instruction Tuning

Enhancing Advanced Visual Reasoning Ability of Large Language Models

A-VL: Adaptive Attention for Large Vision-Language Models

Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model

Enhancing Zero-shot Counting via Language-guided Exemplar Learning

Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models

CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations

CoLLaVO: Crayon Large Language and Vision mOdel

CLIP-Count: Towards Text-Guided Zero-Shot Object Counting

Enhancing Large Vision Language Models with Self-Training on Image Comprehension