The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective

Zhen Qin,Daoyuan Chen,Wenhao Zhang,Liuyi Yao,Yilun Huang,Bolin Ding,Yaliang Li,Shuiguang Deng

2024-08-05

Abstract:The rapid development of large language models (LLMs) has been witnessed in recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the modality from text to a broader spectrum of domains, attracting widespread attention due to the broader range of application scenarios. As LLMs and MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, the importance of data is receiving increasingly widespread attention and recognition. Tracing and analyzing recent data-oriented works for MLLMs, we find that the development of models and data is not two separate paths but rather interconnected. On the one hand, vaster and higher-quality data contribute to better performance of MLLMs; on the other hand, MLLMs can facilitate the development of data. The co-development of multi-modal data and MLLMs requires a clear view of 1) at which development stages of MLLMs specific data-centric approaches can be employed to enhance certain MLLM capabilities, and 2) how MLLMs, utilizing those capabilities, can contribute to multi-modal data in specific roles. To promote the data-model co-development for MLLM community, we systematically review existing works related to MLLMs from the data-model co-development perspective. A regularly maintained project associated with this survey is accessible at <a class="link-external link-https" href="https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper aims to explore the synergy between Multimodal Large Language Models (MLLMs) and data, providing a systematic review of related research from the perspective of the co-evolution of data and models. Specifically:

1. **Main Objectives**: By analyzing the development process of multimodal data and MLLMs, the paper clearly identifies how specific data-centric approaches enhance the capabilities of MLLMs at different stages of development, and conversely, how well-trained models promote the improvement of multimodal data through these capabilities.

2. **Research Questions**:
- RQ1: At which stages in the lifecycle of MLLMs can specific data-centric approaches be applied to enhance particular MLLM capabilities?
- RQ2: In what ways can models assist in the curation of multimodal data? What specific capabilities do models utilize in these scenarios?

3. **Contributions**:
- Proposes a new classification system that emphasizes the synergy between multimodal data and MLLMs, aiming to uncover the mutual benefits in the development of data and models.
- Systematically reviews the rapid advancements in the field of MLLMs, elucidating which MLLM capabilities can be enhanced through specific data-centric approaches and how well-trained models, in turn, nurture multimodal data.
- Provides a roadmap for future directions, focusing on the internal interactions between data and models, offering inspiration and guidance for academic researchers and industry practitioners.

4. **Core Insights**: The co-evolution of data and models holds great potential for MLLMs, but current research on this topic is insufficient. The paper fills the gap in the existing literature on the co-evolution of data and models by proposing a new perspective and classification system.