Abstract:Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a sentence. To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans. To achieve it, we first formulate Region-Centric tasks and construct a large-scale dataset, MedRegInstruct, to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a Region-Aware medical MLLM, MedRegA, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. Our project page is <a class="link-external link-https" href="https://medrega.github.io" rel="external noopener nofollow">this https URL</a>.

Application and Development of AI Intelligent Hospital Guidance System Based on Multimodal Large Language Model (MLLM)

Design and Implementation of AI Intelligent Hospital Guidance System Based on Large Language Model - A Case Study of Henan Children's Hospital

AI Hospital: Interactive Evaluation and Collaboration of LLMs As Intern Doctors for Clinical Diagnosis

Large Language Models Illuminate a Progressive Pathway to Artificial Intelligent Healthcare Assistant

AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review

Guiding IoT-Based Healthcare Alert Systems with Large Language Models

AI Assisted Clinical Diagnosis & Treatment, and Development Strategy

Towards Human-AI Collaboration in Healthcare: Guided Deferral Systems with Large Language Models

From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice

Health-LLM: Personalized Retrieval-Augmented Disease Prediction System

A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine

Large language models in health care: Development, applications, and challenges

Demystifying Large Language Models for Medicine: A Primer

An Automatic Clinical Model for Chinese Medicine Based on Interpretable Machine Learning Algorithms

Architecture design and application practice of clinical intelligent application platform

Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator

Systems Engineering Issues for Industry Applications of Large Language Model

PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation

Application and technology of an open source AI large language model in the medical field

Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks