Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Elisa Warner,Joonsang Lee,William Hsu,Tanveer Syeda-Mahmood,Charles Kahn,Olivier Gevaert,Arvind Rao

2024-01-20

Abstract:Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of "big data" in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily explores the application of multimodal machine learning in medical image analysis and clinical decision support systems, as well as the challenges it faces, and proposes directions for future development. Specifically, this review paper focuses on the application of Multimodal Machine Learning (MML) in the biomedical field, particularly its profound impact on medical image analysis and clinical decision support systems. The article emphasizes key challenges such as multimodal data representation, fusion, transformation, alignment, and collaborative learning, and discusses the transformative potential of these multimodal models in clinical prediction.

The authors point out several major issues currently present:
1. **Representation**: How to geometrically represent data from different modalities while maintaining their intrinsic relationships.
2. **Fusion**: How to effectively integrate data from multiple modalities into a single predictive model.
3. **Transformation**: How to map one modality to another.
4. **Alignment**: How to align two different modalities spatially or temporally.
5. **Collaborative Learning**: How to use one modality to assist the learning process of another modality.

Additionally, the paper highlights ongoing challenges such as data bias and the scarcity of "big data," and discusses principle-based innovations and collaborative efforts to further the mission of seamlessly integrating multimodal machine learning models into biomedical practice. In summary, this paper aims to introduce current and novel approaches to address each multimodal challenge and envisions the future development prospects and potential advancements of artificial intelligence in the biomedical field.