The Revolution of Multimodal Large Language Models: A Survey

Davide Caffagni,Federico Cocchi,Luca Barsellotti,Nicholas Moratelli,Sara Sarto,Lorenzo Baraldi,Lorenzo Baraldi,Marcella Cornia,Rita Cucchiara

2024-06-07

Abstract:Connecting text and visual modalities plays an essential role in generative intelligence. For this reason, inspired by the success of large language models, significant research efforts are being devoted to the development of Multimodal Large Language Models (MLLMs). These models can seamlessly integrate visual and textual modalities, while providing a dialogue-based interface and instruction-following capabilities. In this paper, we provide a comprehensive review of recent visual-based MLLMs, analyzing their architectural choices, multimodal alignment strategies, and training techniques. We also conduct a detailed analysis of these models across a wide range of tasks, including visual grounding, image generation and editing, visual understanding, and domain-specific applications. Additionally, we compile and describe training datasets and evaluation benchmarks, conducting comparisons among existing models in terms of performance and computational requirements. Overall, this survey offers a comprehensive overview of the current state of the art, laying the groundwork for future MLLMs.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language,Multimedia

What problem does this paper attempt to address?

This paper focuses on the development of Multimodal Large Language Models (MLLMs). These models integrate visual and textual information and provide a dialogue-based interface and instruction-following capability. The paper provides a comprehensive review of recent visual-based MLLMs, analyzing their architecture choices, multimodal alignment strategies, and training techniques. The authors also analyze the performance of these models on various tasks, including visual localization, image generation and editing, visual understanding, and domain-specific applications. Additionally, they compile training datasets and evaluation benchmarks, comparing the performance and computational requirements of existing models. This survey aims to provide an overview of the current research status and lay the foundation for future developments of MLLMs. The paper particularly emphasizes the architecture of the models, training methods, and the tasks they are designed to perform.