Multimodal Medical Disease Classification with LLaMA II

Christian Gapp,Elias Tappeiner,Martin Welk,Rainer Schubert

2024-12-02

Abstract:Medical patient data is always multimodal. Images, text, age, gender, histopathological data are only few examples for different modalities in this context. Processing and integrating this multimodal data with deep learning based methods is of utmost interest due to its huge potential for medical procedure such as diagnosis and patient treatment planning. In this work we retrain a multimodal transformer-based model for disease classification. To this end we use the text-image pair dataset from OpenI consisting of 2D chest X-rays associated with clinical reports. Our focus is on fusion methods for merging text and vision information extracted from medical datasets. Different architecture structures with a LLaMA II backbone model are tested. Early fusion of modality specific features creates better results with the best model reaching 97.10% mean AUC than late fusion from a deeper level of the architecture (best model: 96.67% mean AUC). Both outperform former classification models tested on the same multimodal dataset. The newly introduced multimodal architecture can be applied to other multimodal datasets with little effort and can be easily adapted for further research, especially, but not limited to, the field of medical AI.

Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to solve the problem of disease classification for multimodal data in the medical field. Specifically, the researchers utilized a multimodal dataset containing text (clinical reports) and images (2D chest X - rays) to explore how to effectively integrate these different types of medical patient data in order to improve the accuracy of disease diagnosis. By using the multimodal Transformer model based on LLaMA II, the paper attempts to improve the existing disease classification methods, especially proposing early - fusion, late - fusion and hybrid - fusion strategies in terms of fusing text and visual information. Eventually, the research results show that the early - fusion strategy achieved an average AUC value of approximately 97.10% on the test set, outperforming the existing classification models. This achievement is not only of great value for medical diagnosis but also provides strong support for future patient treatment plans.

Multimodal Medical Disease Classification with LLaMA II

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Medical Multimodal Classifiers Under Scarce Data Condition

Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data

Multimodal Fusion Learning with Dual Attention for Medical Imaging

Multi-modal Medical Data Fusion using Deep Learning

Automated Ensemble Multimodal Machine Learning for Healthcare

Disease Classification Model Based on Multi-Modal Feature Fusion

Multimodal medical image fusion and classification using deep learning techniques

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

Multi-modal Multi-instance Learning Using Weakly Correlated Histopathological Images and Tabular Clinical Information

Multimodal Medical Imaging Using Modern Deep Learning Approaches

Multimodal risk prediction with physiological signals, medical images and clinical notes

Multi-task fusion for improving mammography screening data classification

A Collaborative Approach for the Development and Application of Machine Learning Solutions for CMR-Based Cardiac Disease Classification

Multimodal Artificial Intelligence in Medicine