Abstract:Simulation-based Medical Education (SBME) has been developed as a cost-effective means of enhancing the diagnostic skills of novice physicians and interns, thereby mitigating the need for resource-intensive mentor-apprentice training. However, feedback provided in most SBME is often directed towards improving the operational proficiency of learners, rather than providing summative medical diagnoses that result from experience and time. Additionally, the multimodal nature of medical data during diagnosis poses significant challenges for interns and novice physicians, including the tendency to overlook or over-rely on data from certain modalities, and difficulties in comprehending potential associations between modalities. To address these challenges, we present DiagnosisAssistant, a visual analytics system that leverages historical medical records as a proxy for multimodal modeling and visualization to enhance the learning experience of interns and novice physicians. The system employs elaborately designed visualizations to explore different modality data, offer diagnostic interpretive hints based on the constructed model, and enable comparative analyses of specific patients. Our approach is validated through two case studies and expert interviews, demonstrating its effectiveness in enhancing medical training.

What problem does this paper attempt to address?

The main problem this paper attempts to address is to enhance the learning experience and capabilities of interns and junior doctors in medical diagnosis. Specifically, the paper focuses on the following issues: 1. **Limitations of Simulation-Based Medical Education (SBME)**: - Current SBME primarily focuses on improving the operational skills of trainees rather than providing comprehensive medical diagnosis based on experience and time. - The multimodal nature of medical data (such as lab test results, imaging images, and medical texts) poses challenges for interns and junior doctors, who may overlook or overly rely on certain modalities of data, making it difficult to understand the potential correlations between different modalities. 2. **Practical Challenges for Interns and Junior Doctors**: - Interns and junior doctors face communication barriers during the actual diagnostic process and may not have enough time or opportunities to fully discuss diagnostic decision-making processes with colleagues. - Lack of experience leads interns and junior doctors to possibly rely too much on textbooks, ignoring the actual situations in clinical practice. - Lack of opportunities to practice on real patients, as many patients are unwilling to be diagnosed by interns due to concerns about misdiagnosis. 3. **Shortcomings of Existing Methods**: - Existing multimodal models, although performing well in specific tasks, have complex architectures that make their internal working mechanisms difficult to understand. - Existing explanation methods (such as post-hoc explanation techniques) are not yet mature enough in the medical field to fully reveal the decision-making process of the models. To address these challenges, the paper proposes a visual analysis system named **DiagnosisAssistant**, which uses historical medical records as proxies to enhance the learning experience of interns and junior doctors through multimodal modeling and visualization. The specific goals include: - **Clarifying diagnostic tasks and data situations**: Providing statistical data of multimodal datasets and clearly defining medical diagnostic tasks. - **Developing reliable and interpretable machine learning models**: Utilizing advanced machine learning techniques to capture the diagnostic process and ensure the accuracy and interpretability of the models. - **Transforming the diagnostic process into user-friendly representations**: Creating intuitive representations of the diagnostic process, enabling users to easily understand and use them. - **Demonstrating the performance and contribution of each modality of data**: Helping users understand the role of each modality of data in the final diagnosis, avoiding over-reliance on any single modality. - **Revealing the relationships between different modalities**: Helping users establish connections between different modalities of data, especially in cases of conflicting information. - **Supporting comparative analysis of individual patients**: By comparing typical cases, helping interns and junior doctors improve their diagnostic skills and reduce the risk of misdiagnosis and missed diagnosis. Through these measures, the DiagnosisAssistant system aims to enhance the learning effectiveness and practical capabilities of interns and junior doctors in medical diagnosis.

Leveraging Historical Medical Records as a Proxy via Multimodal Modeling and Visualization to Enrich Medical Diagnostic Learning

Leveraging Historical Medical Records as a Proxy via Multimodal Modeling and Visualization to Enrich Medical Diagnostic Learning

VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Models.

Medillustrator: Improving Retrospective Learning in Physicians' Continuous Medical Education via Multimodal Diagnostic Data Alignment and Representation

Leveraging Interpretable Feature Representations for Advanced Differential Diagnosis in Computational Medicine

Bringing Modern Machine Learning into Clinical Practice Through the Use of Intuitive Visualization and Human–Computer Interaction

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis

Stone Needle: A General Multimodal Large-scale Model Framework towards Healthcare

Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm

A Novel Ensemble Learning Paradigm For Medical Diagnosis With Imbalanced Data

A Collaborative Medical Diagnosis System Without Sharing Patient Data

Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

Multimodal risk prediction with physiological signals, medical images and clinical notes

Beyond Doctors: Future Health Prediction from Multimedia and Multimodal Observations.

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

“My Nose is Running.” “are You Also Coughing?”: Building A Medical Diagnosis Agent with Interpretable Inquiry Logics

How to Leverage Multimodal EHR Data for Better Medical Predictions?

AI-Driven Diagnostic Assistance in Medical Inquiry: Reinforcement Learning Algorithm Development and Validation

Latent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration

Gradient modulated contrastive distillation of low-rank multi-modal knowledge for disease diagnosis

Insight: A Multi-Modal Diagnostic Pipeline using LLMs for Ocular Surface Disease Diagnosis