Abstract:Health Informatics Journal, Volume 30, Issue 4, October-December 2024. Objectives: Machine learning-based analytics over uni-modal medical data has shown considerable promise and is now routinely deployed in diagnostic procedures. However, patient data consists of diverse types of data. By exploiting such data, multimodal approaches promise to revolutionize our ability to provide personalized care. Attempts to combine two modalities in a single diagnostic task have utilized the evolving field of multimodal representation learning (MRL), which learns a shared latent space between related modality samples. This new space can be used to improve the performance of machine-learning-based analytics. So far, however, our understanding of how modalities have been applied in MRL-based medical applications and which modalities are best suited for specific medical tasks is still unclear, as previous reviews have not addressed the medical analytics domain and its unique challenges and opportunities. Instead, this work aims to review the landscape of MRL for medical tasks to highlight opportunities for advancing medical applications. Methods: This paper presents a framework for positioning MRL techniques and medical modalities. More than 1000 papers related to medical analytics were reviewed, positioned, and classified using the proposed framework in the most extensive review to date. The paper further provides an online tool for researchers and developers of medical analytics to dive into the rapidly changing landscape of MRL for medical applications. Results: The main finding is that work in the domain has been sparse: only a few medical informatics tasks have been the target of much MRL-based work, with the overwhelming majority of tasks being diagnostic rather than prognostic. Similarly, numerous potentially compatible information modality combinations are unexplored or under-explored for most medical tasks. Conclusions: There is much to gain from using MRL in many unexplored combinations of medical tasks and modalities. This work can guide researchers working on a specific medical application to identify under-explored modality combinations and identify novel and emerging MRL techniques that can be adapted to the task at hand.

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

A Review on Methods and Applications in Multimodal Deep Learning

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

Recent Advances and Trends in Multimodal Deep Learning: A Review

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodal representation learning for medical analytics - a systematic literature review

Multi-modal Machine Learning in Engineering Design: A Review and Future Directions

Multimodal Machine Learning: A Survey and Taxonomy

Surveying the MLLM Landscape: A Meta-Review of Current Surveys

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Reviewing Multimodal Machine Learning and Its Use in Cardiovascular Diseases Detection

Review of multimodal machine learning approaches in healthcare

Multimodal Sentiment Analysis: A Systematic review of History, Datasets, Multimodal Fusion Methods, Applications, Challenges and Future Directions

Modality Influence in Multimodal Machine Learning

Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

A scoping review on multimodal deep learning in biomedical images and texts