Abstract:Although deep cross-modal hashing methods have shown superiorities for cross-modal retrieval recently, there is a concern about potential data privacy leakage when training the models. Federated learning adopts a distributed machine learning strategy, which can collaboratively train models without leaking local private data. It is a promising technique to support privacy-preserving cross-modal hashing. However, existing federated learning-based cross-modal retrieval methods usually rely on a large number of semantic annotations, which limits the scalability of the retrieval models. Furthermore, they mostly update the global models by aggregating local model parameters, ignoring the differences in the quantity and category of multi-modal data from multiple clients. To address these issues, we propose a Prototype Transfer-based Federated Unsupervised Cross-modal Hashing(PT-FUCH) method for solving the privacy leakage problem in cross-modal retrieval model learning. PT-FUCH protects local private data by exploring unified global prototypes for different clients, without relying on any semantic annotations. Global prototypes are used to guide the local cross-modal hash learning and promote the alignment of the feature space, thereby alleviating the model bias caused by the difference in the distribution of local multi-modal data and improving the retrieval accuracy. Additionally, we design an adaptive cross-modal knowledge distillation to transfer valuable semantic knowledge from modal-specific global models to local prototype learning processes, reducing the risk of overfitting. Experimental results on three benchmark cross-modal retrieval datasets validate that our PT-FUCH method can achieve outstanding retrieval performance when trained under distributed privacy-preserving mode. The source codes of our method are available at https://github.com/exquisite1210/PT-FUCH_P.

Cross-Training with Prototypical Distillation for Improving the Generalization of Federated Learning.

Cross-Training with Multi-View Knowledge Fusion for Heterogenous Federated Learning

FedDGP: Disentangling Global and Personal Models for Federated Learning

Improving Global Generalization and Local Personalization for Federated Learning

Global prototype distillation for heterogeneous federated learning

Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data

FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning

Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains

FedCCRL: Federated Domain Generalization with Cross-Client Representation Learning

Tackling Data Heterogeneity in Federated Learning with Class Prototypes

TARGET: Federated Class-Continual Learning Via Exemplar-Free Distillation

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data

An Enhanced Federated Prototype Learning Method under Domain Shift

Prototype-guided Knowledge Transfer for Federated Unsupervised Cross-modal Hashing

Dynamic heterogeneous federated learning with multi-level prototypes

FedProto: Federated Prototype Learning across Heterogeneous Clients

FedGKD: Towards Heterogeneous Federated Learning via Global Knowledge Distillation

Learning to Generalize in Heterogeneous Federated Networks