Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation

Qian Dai,Dong Wei,Hong Liu,Jinghan Sun,Liansheng Wang,Yefeng Zheng
2024-03-18
Abstract:Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, it is not uncommon that some FL participants only possess a subset of the complete imaging modalities, posing inter-modal heterogeneity as a challenge to effectively training a global model on all participants' data. In addition, each participant would expect to obtain a personalized model tailored for its local data characteristics from the FL in such a scenario. In this work, we propose a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues. Above all, FedMEMA employs an exclusive encoder for each modality to account for the inter-modal heterogeneity in the first place. In the meantime, while the encoders are shared by the participants, the decoders are personalized to meet individual needs. Specifically, a server with full-modal data employs a fusion decoder to aggregate and fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation reversely. Meanwhile, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the encoder parameters. On the other end, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up the information loss due to absent modalities while adapting the representations of present ones. FedMEMA is validated on the BraTS 2020 benchmark for multimodal brain tumor segmentation. Results show that it outperforms various up-to-date methods for multimodal and personalized FL and that its novel designs are effective. Our code is available.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
This paper mainly discusses the problems of the existing federated learning (FL) methods in medical image analysis, and proposes a new framework called Federated Learning with Modality-specific Encoders and Multi-modal Anchors (FedMEMA) to address the cross-modal heterogeneity, especially in multi-modal brain tumor segmentation tasks. Traditional FL methods usually only consider the heterogeneity within the same modality. However, in practice, due to the incomplete imaging modality data from different institutions, the challenge of cross-modal heterogeneity arises. FedMEMA addresses this issue by designing independent encoders for each modality, while maintaining shared encoders and personalized decoders to adapt to the data characteristics of different participants. The server-side possesses the complete multi-modal data, aggregates and fuses the representations from different encoders through the multi-modal fusion decoder, and optimizes the encoders in reverse. In addition, multiple anchors are extracted from the fused multi-modal representation and distributed to the clients. The clients utilize the scaled dot-product attention mechanism to calibrate the representations of their missing modalities, compensating for the loss caused by the lack of modality information, while adapting to the representations of existing modalities. Experimental results demonstrate that FedMEMA outperforms existing methods in multi-modal and personalized FL tasks, proving its effectiveness. In summary, this paper aims to address the challenges of training global models and personalized models in multi-modal medical image analysis caused by the lack of certain imaging modalities from some participants through the FedMEMA framework.