Abstract:We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, addressing the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the shortcomings of Multimodal Large Language Models (MLLMs) in visual representation learning. Specifically, the paper focuses on the following key issues: 1. **Insufficient Design of Visual Components**: Current multimodal models often lack sufficient design in their visual components, disconnecting from visual representation learning research. This gap leads to inaccurate sensory foundations in real-world scenarios. 2. **Limitations of Existing Benchmarks**: Existing multimodal model benchmarks may not provide adequate guidance for real-world applications, especially in cases requiring strong visual foundations. These benchmarks may overly rely on language capabilities while neglecting the evaluation of visual perception. 3. **New Methods for Visual Representation Evaluation**: Traditional methods for evaluating visual representations (such as linear probing and end-to-end fine-tuning) have become saturated and fail to reflect the diverse perceptual challenges of the real world. Therefore, new evaluation protocols need to be explored to better guide the development of future visual representations. ### Main Contributions 1. **Introduction of the Cambrian-1 Model Family**: This is a series of vision-centric multimodal large language models designed to evaluate different visual representations through visual instruction tuning. 2. **Development of the New Vision-Centric Benchmark CV-Bench**: This transforms traditional vision benchmarks into a visual question-answering format to better assess visual representations in multimodal environments. 3. **Proposal of the Spatial Vision Aggregator (SVA)**: This is a dynamic and spatially-aware connector design that integrates high-resolution visual features with large language models while reducing the number of tokens. 4. **Curation of High-Quality Visual Instruction Tuning Data**: High-quality visual instruction tuning data is collected and curated from public resources, emphasizing the importance of data source balance and distribution proportions. 5. **Detailed Training and Evaluation Recipes**: Model weights, open-source code, datasets, and detailed training and evaluation recipes are provided to support community research and development. ### Conclusion Through these contributions, Cambrian-1 not only achieves state-of-the-art performance on multiple benchmarks but also provides comprehensive open resources for further research in visual representation learning and multimodal systems. It is hoped that this work will inspire and accelerate progress in related fields.

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models

NVLM: Open Frontier-Class Multimodal LLMs

Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

CaMML: Context-Aware Multimodal Learner for Large Models

LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge

Demonstrative Instruction Following in Multimodal LLMs Via Integrating Low-Rank Adaptation with Ensemble Learning

Towards Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Instruction Tuning-free Visual Token Complement for Multimodal LLMs

MMBench: Is Your Multi-modal Model an All-around Player?

Improved Baselines with Visual Instruction Tuning

InfMLLM: A Unified Framework for Visual-Language Tasks.

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks