Abstract:Expanding the long-context capabilities of Multi-modal Large Language Models~(MLLMs) is crucial for video understanding, high-resolution image understanding, and multi-modal agents. This involves a series of systematic optimizations, including model architecture, data construction and training strategy, particularly addressing challenges such as \textit{degraded performance with more images} and \textit{high computational costs}. In this paper, we adapt the model architecture to a hybrid of Mamba and Transformer blocks, approach data construction with both temporal and spatial dependencies among multiple images and employ a progressive training strategy. The released model \textbf{LongLLaVA}~(\textbf{Long}-Context \textbf{L}arge \textbf{L}anguage \textbf{a}nd \textbf{V}ision \textbf{A}ssistant) is the first hybrid MLLM, which achieved a better balance between efficiency and effectiveness. LongLLaVA not only achieves competitive results across various benchmarks, but also maintains high throughput and low memory consumption. Especially, it could process nearly a thousand images on a single A100 80GB GPU, showing promising application prospects for a wide range of tasks.

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve The paper aims to address the issues of performance degradation and high computational costs encountered by multimodal large language models (MLLMs) when handling multi-image scenarios. Specifically, the paper focuses on the following aspects: 1. **Performance Degradation in Multi-Image Scenarios**: Existing MLLMs often experience significant performance drops when processing multiple images, especially in understanding long videos, high-resolution images, and making decisions based on more historical messages. 2. **High Computational Costs**: As the number of input images increases, the input length of the model rapidly increases, leading to a substantial rise in computational demands and memory usage, which affects user experience and increases service costs. 3. **Extending the Contextual Capability of Multimodal Models**: To improve user experience and broaden the application scope of MLLMs, it is necessary to extend the contextual capabilities of these models to handle longer videos, higher resolution images, and make decisions based on more historical messages. ### Solution To address the above issues, the paper proposes a systematic solution called **LongLLaVA**, which is achieved through optimizations in the following three aspects: 1. **Multimodal Architecture**: - Adopts a hybrid architecture combining Transformer and Mamba blocks. - Proposes an efficient image representation method by compressing image tokens through 2D pooling, significantly reducing computational costs while maintaining performance. 2. **Data Construction**: - Designs a unique data format that enables the model to distinguish the spatiotemporal dependencies between images. 3. **Training Strategy**: - Employs a three-stage multimodal adaptation training strategy: single-image alignment, single-image instruction tuning, and multi-image instruction tuning, gradually enhancing the model's ability to handle long multimodal contexts. ### Main Contributions - **Proposing LongLLaVA**: This is a solution optimized through data construction, training strategy, and multimodal architecture, effectively balancing performance and efficiency. To the best of the authors' knowledge, this is the first hybrid architecture used for MLLMs. - **Outstanding Multimodal Long Context Understanding Capability**: LongLLaVA excels in retrieval, ranking, and counting tasks, leading in VNBench (Zhao et al., 2024e), and is capable of processing nearly 1000 images on a single 80GB GPU, demonstrating the potential for wide task applications. - **Open Source**: For transparency and community research, the authors will open source all models, codes, and datasets related to LongLLaVA. Through these improvements, LongLLaVA not only achieves competitive results in various benchmarks but also maintains high throughput and low memory consumption, showcasing great potential in multimodal long context understanding.

LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning

INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model

LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer

Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification

ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models

AVG-LLaVA: A Large Multimodal Model with Adaptive Visual Granularity

TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations

InfMLLM: A Unified Framework for Visual-Language Tasks.

VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

LongVILA: Scaling Long-Context Visual Language Models for Long Videos

LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges

ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2

Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM

HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models

VoCo-LLaMA: Towards Vision Compression with Large Language Models

LLaVA-OneVision: Easy Visual Task Transfer