Abstract:Multi-modal large language models (MLLMs) have shown impressive capabilities as a general-purpose interface for various visual and linguistic tasks. However, building a unified MLLM for multi-task learning in the medical field remains a thorny challenge. To mitigate the tug-of-war problem of multi-modal multi-task optimization in MLLMs, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities. In this paper, we introduce Uni-Med, a novel medical generalist foundation model which consists of a universal visual feature extraction module, a connector mixture-of-experts (CMoE) module, and an LLM. Benefiting from the proposed CMoE that leverages a well-designed router with a mixture of projection experts at the connector, Uni-Med achieves efficient solution to the tug-of-war problem and can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification. To the best of our knowledge, Uni-Med is the first effort to tackle multi-task interference at the connector in MLLMs. Extensive ablation experiments validate the effectiveness of introducing CMoE under any configuration, with up to an average 8% performance gains. We further provide interpretation analysis of the tug-of-war problem from the perspective of gradient optimization and parameter statistics. Compared to previous state-of-the-art medical MLLMs, Uni-Med achieves competitive or superior evaluation metrics on diverse tasks. Code and resources are available at <a class="link-external link-https" href="https://github.com/tsinghua-msiip/Uni-Med" rel="external noopener nofollow">this https URL</a>.

Measuring the Non-Compositionality of Multiword Expressions

A New Multiword Expression Metric and Its Applications

Automatic Extraction of Multiword Expressions Combining Statistical and Similarity Approaches

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Contextual Compositionality Detection with External Knowledge Bases andWord Embeddings

Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE

CMMLU: Measuring massive multitask language understanding in Chinese

Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models

Parsing Models for Identifying Multiword Expressions

Detecting Unseen Multiword Expressions in American Sign Language

Multilingual Multiword Expressions

Unsupervised Paraphrasing of Multiword Expressions

Detecting Idiomatic Multiword Expressions in Clinical Terminology using Definition-Based Representation Learning

Measuring Word Polysemousness And Sense Granularity At A Language Level

Construction and Application of the Knowledge Base of Chinese Multi-word Expressions

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptation

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens

MAF - A General Matching and Alignment Framework for Multimodal Named Entity Recognition.

Bilingual Multi-word Expressions, Multiple-correspondence, and Their Cultivation from Parallel Patents: the Chinese-English Case