Abstract:Significant advancements has recently been achieved in the field of multi-modal large language models (MLLMs), demonstrating their remarkable capabilities in understanding and reasoning across diverse tasks. However, these models are often trained for specific tasks and rely on task-specific input-output formats, limiting their applicability to a broader range of tasks. This raises a fundamental question: Can we develop a unified approach to represent and handle different multi-modal tasks to maximize the generalizability of MLLMs? In this paper, we propose UnifiedMLLM, a comprehensive model designed to represent various tasks using a unified representation. Our model exhibits strong capabilities in comprehending the implicit intent of user instructions and preforming reasoning. In addition to generating textual responses, our model also outputs task tokens and grounding tokens, serving as indicators of task types and task granularity. These outputs are subsequently routed through the task router and directed to specific expert models for task completion. To train our model, we construct a task-specific dataset and an 100k multi-task dataset encompassing complex scenarios. Employing a three-stage training strategy, we equip our model with robust reasoning and task processing capabilities while preserving its generalization capacity and knowledge reservoir. Extensive experiments showcase the impressive performance of our unified representation approach across various tasks, surpassing existing methodologies. Furthermore, our approach exhibits exceptional scalability and generality. Our code, model, and dataset will be available at \url{<a class="link-external link-https" href="https://github.com/lzw-lzw/UnifiedMLLM" rel="external noopener nofollow">this https URL</a>}.

MM-LLMs: Recent Advances in MultiModal Large Language Models

MM-LLMs: Recent Advances in MultiModal Large Language Models

A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety

Efficient Multimodal Large Language Models: A Survey

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

A Survey on Multimodal Large Language Models

LLMs Meet Multimodal Generation and Editing: A Survey

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

Personalized Multimodal Large Language Models: A Survey

A Survey of Multimodal Large Language Model from A Data-centric Perspective

From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding

A Review of Multi-Modal Large Language and Vision Models

Continuous or Discrete, That is the Question: A Survey on Large Multi-Modal Models from the Perspective of Input-Output Space Extension

Surveying the MLLM Landscape: A Meta-Review of Current Surveys

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

Understanding the Role of LLMs in Multimodal Evaluation Benchmarks

A Survey on Benchmarks of Multimodal Large Language Models

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers