Abstract:Medical image reports are integral to clinical decision-making and patient management. Despite their importance, the confidentiality and private nature of medical data pose significant issues for the sharing and analysis of medical image data. This paper addresses these concerns by introducing a multimodal federated learning-based methodology for medical image reporting. This methodology harnesses distributed computing for co-training models across various medical institutions. Under the federated learning framework, every medical institution is capable of training the model locally and aggregating the updated model parameters to curate a top-tier medical image report model. Initially, we advocate for an architecture facilitating multimodal federated learning, including model creation, parameter consolidation, and algorithm enhancement steps. In the model selection phase, we introduce a deep learning-based strategy that utilizes multimodal data for training to produce medical image reports. In the parameter aggregation phase, the federal average algorithm is applied to amalgamate model parameters trained by each institution, which leads to a comprehensive global model. In addition, we introduce an evidence-based optimization algorithm built upon the federal average algorithm. The efficacy of the proposed architecture and scheme is showcased through a series of experiments. Our experimental results validate the proficiency of the proposed multimodal federated learning approach in generating medical image reports. Compared to conventional centralized learning methods, our proposal not only enhances the protection of patient confidentiality but also enriches the accuracy and overall quality of medical image reports. Through this research, we offer a novel solution for the privacy issues linked with the sharing and analyzing of medical data. Expected to assume a crucial role in medical image report generation and other medical applications, the multimodal federated learning method is set to deliver more precise, efficient, and privacy-secured medical services for healthcare professionals and patients.

Modality Bank: Learn multi-modality images across data centers without sharing medical data

Medical Multi-Modal Image Transformation with Modality Code Awareness

Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation

Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without Sharing Private Information

Unified Multi-Modal Image Synthesis for Missing Modality Imputation

A Survey of Cross-Modality Brain Image Synthesis

MRIS: A Multi-modal Retrieval Approach for Image Synthesis on Diverse Modalities

MDA: An Interpretable and Scalable Multi-Modal Fusion under Missing Modalities and Intrinsic Noise Conditions

FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology

Modality-Agnostic Learning for Medical Image Segmentation Using Multi-modality Self-distillation

Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction

Multimodal image synthesis based on disentanglement representations of anatomical and modality specific features, learned using uncooperative relativistic GAN

FedMed-GAN: Federated Domain Translation on Unsupervised Cross-Modality Brain Image Synthesis

MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks

Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach

AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image Synthesis

Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities

Medical report generation based on multimodal federated learning

Multi-modal Modality-masked Diffusion Network for Brain MRI Synthesis with Random Modality Missing

Inter-modality Dependence Induced Data Recovery for MCI Conversion Prediction

Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration