Abstract:Significant advancements has recently been achieved in the field of multi-modal large language models (MLLMs), demonstrating their remarkable capabilities in understanding and reasoning across diverse tasks. However, these models are often trained for specific tasks and rely on task-specific input-output formats, limiting their applicability to a broader range of tasks. This raises a fundamental question: Can we develop a unified approach to represent and handle different multi-modal tasks to maximize the generalizability of MLLMs? In this paper, we propose UnifiedMLLM, a comprehensive model designed to represent various tasks using a unified representation. Our model exhibits strong capabilities in comprehending the implicit intent of user instructions and preforming reasoning. In addition to generating textual responses, our model also outputs task tokens and grounding tokens, serving as indicators of task types and task granularity. These outputs are subsequently routed through the task router and directed to specific expert models for task completion. To train our model, we construct a task-specific dataset and an 100k multi-task dataset encompassing complex scenarios. Employing a three-stage training strategy, we equip our model with robust reasoning and task processing capabilities while preserving its generalization capacity and knowledge reservoir. Extensive experiments showcase the impressive performance of our unified representation approach across various tasks, surpassing existing methodologies. Furthermore, our approach exhibits exceptional scalability and generality. Our code, model, and dataset will be available at \url{<a class="link-external link-https" href="https://github.com/lzw-lzw/UnifiedMLLM" rel="external noopener nofollow">this https URL</a>}.

Machine-Created Universal Language for Cross-lingual Transfer

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

MULE: Multimodal Universal Language Embedding

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

MulCS: Towards a Unified Deep Representation for Multilingual Code Search

Unified Model Learning for Various Neural Machine Translation

On Learning Universal Representations Across Languages.

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Unified Language Model Pre-training for Natural Language Understanding and Generation

MURAL: Multimodal, Multitask Retrieval Across Languages

Unsupervised Neural Machine Translation with Cross-Lingual Language Representation Agreement

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models

VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

FC-MTLF: A Fine- and Coarse-grained Multi-Task Learning Framework for Cross-Lingual Spoken Language Understanding.

Multilingual Multimodal Learning with Machine Translated Text

mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?

Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation

Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation