Abstract:Cancer research encompasses data across various scales, modalities, and resolutions, from screening and diagnostic imaging to digitized histopathology slides to various types of molecular data and clinical records. The integration of these diverse data types for personalized cancer care and predictive modeling holds the promise of enhancing the accuracy and reliability of cancer screening, diagnosis, and treatment. Traditional analytical methods, which often focus on isolated or unimodal information, fall short of capturing the complex and heterogeneous nature of cancer data. The advent of deep neural networks has spurred the development of sophisticated multimodal data fusion techniques capable of extracting and synthesizing information from disparate sources. Among these, Graph Neural Networks (GNNs) and Transformers have emerged as powerful tools for multimodal learning, demonstrating significant success. This review presents the foundational principles of multimodal learning including oncology data modalities, taxonomy of multimodal learning, and fusion strategies. We delve into the recent advancements in GNNs and Transformers for the fusion of multimodal data in oncology, spotlighting key studies and their pivotal findings. We discuss the unique challenges of multimodal learning, such as data heterogeneity and integration complexities, alongside the opportunities it presents for a more nuanced and comprehensive understanding of cancer. Finally, we present some of the latest comprehensive multimodal pan-cancer data sources. By surveying the landscape of multimodal data integration in oncology, our goal is to underline the transformative potential of multimodal GNNs and Transformers. Through technological advancements and the methodological innovations presented in this review, we aim to chart a course for future research in this promising field. This review may be the first that highlights the current state of multimodal modeling applications in cancer using GNNs and transformers, presents comprehensive multimodal oncology data sources, and sets the stage for multimodal evolution, encouraging further exploration and development in personalized cancer care.

Multimodal data integration for oncology in the era of deep neural networks: a review

Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review

Digital pathology and multimodal learning on oncology data

Multimodal Data Integration for Precision Oncology: Challenges and Future Directions

Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes

Harnessing multimodal data integration to advance precision oncology

Multimodal Deep Learning for Cancer Survival Prediction: A Review

A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification

Artificial intelligence (AI) and machine learning (ML) in precision oncology: a review on enhancing discoverability through multiomics integration

Deep Learning Based Multimodal Biomedical Data Fusion: an Overview and Comparative Review

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis

Quantifying the advantage of multimodal data fusion for survival prediction in cancer patients

A review of cancer data fusion methods based on deep learning

The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review

Integrated Multi-Omics Analyses in Oncology: A Review of Machine Learning Methods and Tools

Multimodal CustOmics: A Unified and Interpretable Multi-Task Deep Learning Framework for Multimodal Integrative Data Analysis in Oncology

Multimodal deep learning for biomedical data fusion: a review

Applications of Multimodal Artificial Intelligence in Non-Hodgkin Lymphoma B Cells

Integration of Multimodal Data from Disparate Sources for Identifying Disease Subtypes

Multi-omics based artificial intelligence for cancer research