Abstract:The analysis of 3D medical images is crucial for modern healthcare, yet traditional task-specific models are becoming increasingly inadequate due to limited generalizability across diverse clinical scenarios. Multimodal large language models (MLLMs) offer a promising solution to these challenges. However, existing MLLMs have limitations in fully leveraging the rich, hierarchical information embedded in 3D medical images. Inspired by clinical practice, where radiologists focus on both 3D spatial structure and 2D planar content, we propose Med-2E3, a novel MLLM for 3D medical image analysis that integrates 3D and 2D encoders. To aggregate 2D features more effectively, we design a Text-Guided Inter-Slice (TG-IS) scoring module, which scores the attention of each 2D slice based on slice contents and task instructions. To the best of our knowledge, Med-2E3 is the first MLLM to integrate both 3D and 2D features for 3D medical image analysis. Experiments on a large-scale, open-source 3D medical multimodal benchmark demonstrate that Med-2E3 exhibits task-specific attention distribution and significantly outperforms current state-of-the-art models, with a 14% improvement in report generation and a 5% gain in medical visual question answering (VQA), highlighting the model's potential in addressing complex multimodal clinical tasks. The code will be released upon acceptance.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to address the limitations of existing 3D medical image analysis models when dealing with complex multi - modal clinical tasks. Specifically, traditional task - specific models have limited generalization ability in different clinical scenarios, and the existing multi - modal large language models (MLLMs) fail to fully utilize the rich hierarchical information in 3D medical images. To solve these problems, the authors propose a new model, Med - 2E3. #### Specific problems: 1. **Limitations of traditional models**: When processing 3D medical images, traditional models usually use only a single perspective (such as using only a 3D encoder or a 2D encoder) and cannot capture the global spatial structure and local planar content simultaneously. 2. **Deficiencies of existing MLLMs**: Existing MLLMs mainly focus on 2D medical images, have relatively limited support for 3D medical images, and fail to effectively combine 3D and 2D features for comprehensive analysis. 3. **Requirements for cross - modal tasks**: In modern medical diagnosis and treatment, the application of 3D medical images is becoming more and more widespread, especially in tasks such as report generation and surgical planning, which require strong feature extraction and information processing capabilities. ### Med - 2E3's solutions: To overcome the above problems, Med - 2E3 introduces the following innovations: 1. **Integrating 3D and 2D encoders**: Med - 2E3 combines 3D and 2D encoders to fully utilize the global spatial structure and local planar content in 3D medical images. 2. **Text - guided slice - scoring module (TG - IS)**: A text - guided slice - scoring module is designed to calculate attention scores according to the content of each slice and the task instructions, thus aggregating 2D features more effectively. 3. **Enhanced 3D feature representation**: By weighted aggregation of 2D features and combination with 3D features, the overall representation ability of 3D medical images is enhanced. These innovations make Med - 2E3 perform excellently in handling complex 3D medical multi - modal tasks, especially in report generation and medical visual question answering (VQA) tasks, significantly outperforming existing models. ### Summary: Med - 2E3 solves the limitations of existing models in processing 3D medical images by integrating 3D and 2D encoders and introducing a text - guided slice - scoring module, and improves the performance of the model in complex multi - modal clinical tasks.

Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model

M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Models

MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training

MOSS-MED: Medical Multimodal Model Serving Medical Image Analysis

MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation

Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE

Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm

Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare

ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences

Towards Evaluating and Building Versatile Large Language Models for Medicine

Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding

PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging

BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities

Language-Image Models with 3D Understanding

E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model

Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models

MMedAgent: Learning to Use Medical Tools with Multi-modal Agent

MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation

LLMI3D: Empowering LLM with 3D Perception from a Single 2D Image