Abstract:Personalized medicine is a groundbreaking healthcare framework for the $21^{st}$ century, tailoring medical treatments to individuals based on unique clinical characteristics, including diverse medical imaging modalities. Given the significant differences among these modalities due to distinct underlying imaging principles, generalization in multi-modal medical image tasks becomes substantially challenging. Previous methods addressing multi-modal generalization rarely consider personalization, primarily focusing on common anatomical information. This paper aims to bridge multi-modal generalization with the concept of personalized medicine. Specifically, we propose a novel approach to derive a tractable form of the underlying personalized invariant representation $\mathbb{X}_h$ by leveraging individual-level constraints and a learnable biological prior. We demonstrate the feasibility and benefits of learning a personalized $\mathbb{X}_h$, showing that this representation is highly generalizable and transferable across various multi-modal medical tasks. Our method is rigorously validated on medical imaging modalities emphasizing both physical structure and functional information, encompassing a range of tasks that require generalization. Extensive experimental results consistently show that our approach significantly improves performance across diverse scenarios, confirming its effectiveness.

What problem does this paper attempt to address?

This paper attempts to address the issue of personalized generalization in multimodal medical imaging. Specifically, it focuses on how to achieve effective generalization across different medical imaging modalities (such as MRI, CT, and PET), especially in cases where certain modalities may be unavailable due to economic or physical constraints, and how to use existing modality information to infer missing modality information. Existing methods mainly focus on general anatomical structure information, neglecting individual differences. Therefore, this paper proposes a new approach by introducing a personalized invariant representation $X_h$ to bridge the concepts of multimodal generalization and personalized medicine. ### Main Issues 1. **Multimodal Generalization Challenge**: Due to significant differences between different imaging modalities, generalization in multimodal medical image tasks becomes very difficult. 2. **Personalization Needs**: Existing methods rarely consider personalization when dealing with multimodal generalization, mainly focusing on general anatomical structure information and ignoring individual differences. 3. **Cross-Modal Information Transfer**: How to effectively infer information from other modalities when only partial modality information is available to support personalized medicine. ### Solution The paper proposes a new approach to address the above issues through the following steps: 1. **Personalized Invariant Representation $X_h$**: Using individual-level constraints and learnable biological priors $O$, derive an operable personalized invariant representation $X_h$. 2. **Multimodal Generalization**: Validate that the learned personalized $X_h$ has high generalization and transferability in various multimodal medical tasks. 3. **Experimental Validation**: Conduct experiments on various medical imaging modalities to demonstrate that the proposed method significantly improves performance in different scenarios. ### Experimental Results - **Modality Conversion Task**: On the BRATS23 dataset, the proposed method significantly outperforms existing 2D and 3D generation methods in the modality conversion tasks from T1 to T2 and T2 to FLAIR. - **Missing Modality Segmentation Task**: On the BRATS18 dataset, the proposed method shows better performance than existing methods under different numbers of missing modality settings, especially in the SSIM metric. ### Conclusion By introducing a personalized invariant representation $X_h$, the paper successfully addresses the issue of personalized generalization in multimodal medical imaging. Experimental results show that the proposed method not only excels in modality conversion tasks but also significantly improves performance in missing modality segmentation tasks, verifying its effectiveness and superiority.

Personalize to generalize: Towards a universal medical multi-modality generalization through personalization

A Survey on Domain Generalization for Medical Image Analysis

Towards Integrated Noninvasive Personalization of Cardiac Electromechanics

Universal Model for 3D Medical Image Analysis

Universal Medical Imaging Model for Domain Generalization with Data Privacy

Multimodal Medical Image Registration Via Common Representations Learning and Differentiable Geometric Constraints

Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation

Towards Generalist Biomedical AI

Learning Personalized Representation for Inverse Problems in Medical Imaging Using Deep Neural Network

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

A Personalized Diagnostic Generation Framework Based on Multi-source Heterogeneous Data

Bilateral Proxy Federated Domain Generalization for Privacy-preserving Medical Image Diagnosis

Artificial Intelligence toward Personalized Medicine

Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

Diversified and Personalized Multi-rater Medical Image Segmentation

Generalized Brain Image Synthesis with Transferable Convolutional Sparse Coding Networks.

Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond

Privacy-Preserving Constrained Domain Generalization via Gradient Alignment

MH-pFLGB: Model Heterogeneous personalized Federated Learning via Global Bypass for Medical Image Analysis

Enhancing Personalized Healthcare via Capturing Disease Severity, Interaction, and Progression

MultiMed: Massively Multimodal and Multitask Medical Understanding