Abstract:Although deep cross-modal hashing methods have shown superiorities for cross-modal retrieval recently, there is a concern about potential data privacy leakage when training the models. Federated learning adopts a distributed machine learning strategy, which can collaboratively train models without leaking local private data. It is a promising technique to support privacy-preserving cross-modal hashing. However, existing federated learning-based cross-modal retrieval methods usually rely on a large number of semantic annotations, which limits the scalability of the retrieval models. Furthermore, they mostly update the global models by aggregating local model parameters, ignoring the differences in the quantity and category of multi-modal data from multiple clients. To address these issues, we propose a Prototype Transfer-based Federated Unsupervised Cross-modal Hashing(PT-FUCH) method for solving the privacy leakage problem in cross-modal retrieval model learning. PT-FUCH protects local private data by exploring unified global prototypes for different clients, without relying on any semantic annotations. Global prototypes are used to guide the local cross-modal hash learning and promote the alignment of the feature space, thereby alleviating the model bias caused by the difference in the distribution of local multi-modal data and improving the retrieval accuracy. Additionally, we design an adaptive cross-modal knowledge distillation to transfer valuable semantic knowledge from modal-specific global models to local prototype learning processes, reducing the risk of overfitting. Experimental results on three benchmark cross-modal retrieval datasets validate that our PT-FUCH method can achieve outstanding retrieval performance when trained under distributed privacy-preserving mode. The source codes of our method are available at https://github.com/exquisite1210/PT-FUCH_P.

Prototype-Based Discriminative Feature Representation for Class-incremental Cross-modal Retrieval.

Learning Disentangled Representation for Cross-Modal Retrieval with Deep Mutual Information Estimation.

Source-enhanced Prototypical Alignment for Single Image 3D Model Retrieval

Semantic-enhanced discriminative embedding learning for cross-modal retrieval

Discriminative Dictionary Learning with Common Label Alignment for Cross-Modal Retrieval.

An Enhanced Incremental Prototype Classifier using Subspace Representation Scheme

Prototypes Sampling Mechanism for Class Incremental Learning

ICA-Proto: Iterative Cross Alignment Prototypical Network for Incremental Few-Shot Relation Classification

Domain Invariant Subspace Learning for Cross-Modal Retrieval.

Adaptive Parametric Prototype Learning for Cross-Domain Few-Shot Classification

Privacy-Enhanced Prototype-based Federated Cross-modal Hashing for Cross-modal Retrieval

Fine-grained Prototypical Voting with Heterogeneous Mixup for Semi-supervised 2D-3D Cross-modal Retrieval

An Enhanced Federated Prototype Learning Method under Domain Shift

Adaptive Cross-Modal Prototypes for Cross-Domain Visual-Language Retrieval

Prototype-guided Knowledge Transfer for Federated Unsupervised Cross-modal Hashing

Multivariate Prototype Representation for Domain-Generalized Incremental Learning

Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains

Federated learning for supervised cross-modal retrieval

Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototype

Rethinking Label-Wise Cross-Modal Retrieval from A Semantic Sharing Perspective