CaMML: Context-Aware Multimodal Learner for Large Models

Yixin Chen,Shuai Zhang,Boran Han,Tong He,Bo Li

2024-06-19

Abstract:In this work, we introduce Context-Aware MultiModal Learner (CaMML), for tuning large multimodal models (LMMs). CaMML, a lightweight module, is crafted to seamlessly integrate multimodal contextual samples into large models, thereby empowering the model to derive knowledge from analogous, domain-specific, up-to-date information and make grounded inferences. Importantly, CaMML is highly scalable and can efficiently handle lengthy multimodal context examples owing to its hierarchical design. Based on CaMML, we have developed two multimodal models, CaMML-7B and CaMML-13B, that have shown exceptional performance across an array of benchmark datasets for multimodal tasks. Remarkably, CaMML-13B achieves the state-of-the-art performance on over ten widely recognized multimodal benchmark datasets, surpassing LLaVA-1.5 (13B) with a noticeable margin, without integration of any external resources. Moreover, we have conducted extensive ablative studies to inspect the inner workings of CaMML and performed qualitative analyses to showcase its effectiveness in handling real-world challenging cases. Code and models are available at: <a class="link-external link-https" href="https://github.com/amazon-science/camml" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the challenges faced by Large Multimodal Models (LMMs) in handling multimodal contextual information. Specifically:

1. **Enhancing the Context-Awareness of Multimodal Models**:
- Current multimodal foundational models, while performing well on various tasks, have their reasoning capabilities limited by the knowledge encoded in the model parameters.
- These models are relatively rigid in design, making it difficult to generalize from contextual examples. For instance, LLaVA-1.5 performs poorly when handling multiple images due to the lack of corresponding instruction-tuning training data.

2. **Improving the Model's Ability to Handle Long Contexts and Heterogeneous Modal Inputs**:
- Multimodal contextual inputs may contain large amounts of data in different modalities, which poses a challenge to existing models.
- A lightweight module is needed to seamlessly integrate these contextual information and enable the model to extract knowledge for context-based reasoning.

To address the above issues, the authors propose a method called **Context-Aware MultiModal Learner (CaMML)**. CaMML is a lightweight module designed to efficiently handle long multimodal contextual samples. In this way, CaMML can significantly enhance the performance of multimodal models across various benchmarks, particularly excelling in complex multi-image and text tasks.