Abstract:Health Informatics Journal, Volume 30, Issue 4, October-December 2024. Objectives: Machine learning-based analytics over uni-modal medical data has shown considerable promise and is now routinely deployed in diagnostic procedures. However, patient data consists of diverse types of data. By exploiting such data, multimodal approaches promise to revolutionize our ability to provide personalized care. Attempts to combine two modalities in a single diagnostic task have utilized the evolving field of multimodal representation learning (MRL), which learns a shared latent space between related modality samples. This new space can be used to improve the performance of machine-learning-based analytics. So far, however, our understanding of how modalities have been applied in MRL-based medical applications and which modalities are best suited for specific medical tasks is still unclear, as previous reviews have not addressed the medical analytics domain and its unique challenges and opportunities. Instead, this work aims to review the landscape of MRL for medical tasks to highlight opportunities for advancing medical applications. Methods: This paper presents a framework for positioning MRL techniques and medical modalities. More than 1000 papers related to medical analytics were reviewed, positioned, and classified using the proposed framework in the most extensive review to date. The paper further provides an online tool for researchers and developers of medical analytics to dive into the rapidly changing landscape of MRL for medical applications. Results: The main finding is that work in the domain has been sparse: only a few medical informatics tasks have been the target of much MRL-based work, with the overwhelming majority of tasks being diagnostic rather than prognostic. Similarly, numerous potentially compatible information modality combinations are unexplored or under-explored for most medical tasks. Conclusions: There is much to gain from using MRL in many unexplored combinations of medical tasks and modalities. This work can guide researchers working on a specific medical application to identify under-explored modality combinations and identify novel and emerging MRL techniques that can be adapted to the task at hand.

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Review of multimodal machine learning approaches in healthcare

Multimodal Machine Learning in Precision Health

Beyond Medical Imaging - A Review of Multimodal Deep Learning in Radiology

Multimodal biomedical AI

Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications

Automated Ensemble Multimodal Machine Learning for Healthcare

Reviewing Multimodal Machine Learning and Its Use in Cardiovascular Diseases Detection

Multimodal Artificial Intelligence in Medicine

Multimodal machine learning in precision health: A scoping review

Multimodal Foundation Models for Medical Imaging - A Systematic Review and Implementation Guidelines

Multimodal representation learning for medical analytics - a systematic literature review

A review on multimodal machine learning in medical diagnostics

The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook

A scoping review on multimodal deep learning in biomedical images and texts

Multimodal Machine Learning: A Survey and Taxonomy