Abstract:In the past year, Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning. However, the extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. Thus, studying efficient and lightweight MLLMs has enormous potential, especially in edge computing scenarios. In this survey, we provide a comprehensive and systematic review of the current state of efficient MLLMs. Specifically, we summarize the timeline of representative efficient MLLMs, research state of efficient structures and strategies, and the applications. Finally, we discuss the limitations of current efficient MLLM research and promising future directions. Please refer to our GitHub repository for more details: <a class="link-external link-https" href="https://github.com/lijiannuist/Efficient-Multimodal-LLMs-Survey" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper aims to address the issues of high resource consumption, expensive training costs, and low inference efficiency in the widespread application of Multimodal Large Language Models (MLLMs) in academia and industry. Specifically, the paper focuses on the following aspects: 1. **Efficient Architecture**: The paper explores how to reduce computational costs by optimizing visual encoders, improving visual-language projectors, and adopting smaller language models. 2. **Efficient Visual Processing**: It investigates how to optimize efficient visual feature extraction strategies, emphasizing maintaining accuracy while improving efficiency, and integrating high-quality visual data to achieve effective cross-modal understanding. 3. **Efficient Language Models**: The discussion includes strategies to enhance the computational efficiency and scalability of language models, exploring the trade-offs between model complexity and performance, and proposing potential ways to balance these competing factors. 4. **Training Methods**: It outlines the methods involved in training efficient MLLMs, including the pre-training phase, instruction fine-tuning phase, and overall training strategies. 5. **Data and Benchmarking**: The paper evaluates the datasets and benchmarks used for assessing multimodal language models, analyzing the trade-offs between dataset size, complexity, and computational costs, and advocates for the development of benchmarks focused on practical application efficiency. 6. **Application Domains**: It examines the practical applications of efficient MLLMs in various fields, with a focus on their potential in resource-intensive tasks such as high-resolution image understanding and medical question answering. In summary, this literature review aims to comprehensively outline the development of efficient MLLMs, provide a classification of efficient MLLM strategies, and compare the performance of existing efficient models to guide future development in this field. Additionally, the authors have created a GitHub repository that compiles the papers mentioned in the survey, organized according to the same classification.

Efficient Multimodal Large Language Models: A Survey

A Survey on Benchmarks of Multimodal Large Language Models

A Survey of Multimodal Large Language Model from A Data-centric Perspective

A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety

Personalized Multimodal Large Language Models: A Survey

Efficient Large Language Models: A Survey

A Survey on Multimodal Large Language Models

Large Multimodal Agents: A Survey

Multimodal Large Language Models: A Survey

Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey

Surveying the MLLM Landscape: A Meta-Review of Current Surveys

A Survey on Evaluation of Multimodal Large Language Models

Multilingual Large Language Models: A Systematic Survey

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

A Survey of Resource-efficient LLM and Multimodal Foundation Models

LLMs Meet Multimodal Generation and Editing: A Survey

The Efficiency Spectrum of Large Language Models: An Algorithmic Survey

Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers