Multimodal Explainability via Latent Shift applied to COVID-19 stratification

Valerio Guarrasi,Lorenzo Tronchin,Domenico Albano,Eliodoro Faiella,Deborah Fazzini,Domiziana Santucci,Paolo Soda

2024-07-22

Abstract:We are witnessing a widespread adoption of artificial intelligence in healthcare. However, most of the advancements in deep learning in this area consider only unimodal data, neglecting other modalities. Their multimodal interpretation necessary for supporting diagnosis, prognosis and treatment decisions. In this work we present a deep architecture, which jointly learns modality reconstructions and sample classifications using tabular and imaging data. The explanation of the decision taken is computed by applying a latent shift that, simulates a counterfactual prediction revealing the features of each modality that contribute the most to the decision and a quantitative score indicating the modality importance. We validate our approach in the context of COVID-19 pandemic using the AIforCOVID dataset, which contains multimodal data for the early identification of patients at risk of severe outcome. The results show that the proposed method provides meaningful explanations without degrading the classification performance.

Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve The paper aims to address the following issues: 1. **Application of Multimodal Data in the Medical Field**: Currently, most deep learning models in the medical field consider only single-modal data, ignoring other available information sources. However, medical diagnosis is inherently multimodal, requiring AI methods capable of handling different modalities of data. 2. **Explainable Artificial Intelligence (XAI)**: Although complex AI models have achieved significant results in many fields, they are often black-box operations, lacking transparency and trustworthiness. Especially in the biomedical field, the interpretability of models is crucial. Therefore, researchers are committed to developing models that can explain their decision-making processes. 3. **Application of Multimodal Explanations in Medicine**: Multimodal models extract more comprehensive information than single-modal models, so their explanations can provide more insights into medical data. Nevertheless, there is currently a lack of interpretable multimodal deep learning models in the biomedical literature. Specifically, the paper proposes a new end-to-end multimodal architecture that combines tabular data and image data, achieving interpretability through joint learning of modality reconstruction and sample classification. This method reveals the contribution of each modality to the decision-making process by simulating counterfactual predictions and provides quantitative scores representing the importance of each modality. Researchers validated this method on the AIforCOVID dataset for the early identification of COVID-19 patients at risk of severe outcomes, showing that the method can provide meaningful explanations without reducing classification performance.

Multimodal Explainability via Latent Shift applied to COVID-19 stratification

Explainable Multi-class Classification of the CAMH COVID-19 Mental Health Data

Representing visual classification as a linear combination of words

Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?

Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classification

Explainability meets uncertainty quantification: Insights from feature-based model fusion on multimodal time series

Ultrasound Diagnosis of COVID-19: Robustness and Explainability

An Explainable AI System for Automated COVID-19 Assessment and Lesion Categorization from CT-scans

COVLIAS 2.0-cXAI: Cloud-Based Explainable Deep Learning System for COVID-19 Lesion Localization in Computed Tomography Scans

A Decision Support System for Diagnosis of COVID-19 from Non-COVID-19 Influenza-like Illness Using Explainable Artificial Intelligence

Multimodal risk prediction with physiological signals, medical images and clinical notes

Generating Post-Hoc Explanation from Deep Neural Networks for Multi-Modal Medical Image Analysis Tasks

A Collaborative Multimodal Learning-Based Framework for COVID-19 Diagnosis

Transparent and Accurate COVID-19 Diagnosis: Integrating Explainable AI with Advanced Deep Learning in CT Imaging

A medical multimodal large language model for future pandemics

Learning by Reasoning: an Explainable Hierarchical Association Regularized Deep Learning Method for Disease Prediction.

Efficient explainable deep learning technique for COVID-19 diagnosis based on computed Tomography scan images of lungs

Latent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration

A multistage multimodal deep learning model for disease severity assessment and early warnings of high-risk patients of COVID-19

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis