Abstract:Although deep cross-modal hashing methods have shown superiorities for cross-modal retrieval recently, there is a concern about potential data privacy leakage when training the models. Federated learning adopts a distributed machine learning strategy, which can collaboratively train models without leaking local private data. It is a promising technique to support privacy-preserving cross-modal hashing. However, existing federated learning-based cross-modal retrieval methods usually rely on a large number of semantic annotations, which limits the scalability of the retrieval models. Furthermore, they mostly update the global models by aggregating local model parameters, ignoring the differences in the quantity and category of multi-modal data from multiple clients. To address these issues, we propose a Prototype Transfer-based Federated Unsupervised Cross-modal Hashing(PT-FUCH) method for solving the privacy leakage problem in cross-modal retrieval model learning. PT-FUCH protects local private data by exploring unified global prototypes for different clients, without relying on any semantic annotations. Global prototypes are used to guide the local cross-modal hash learning and promote the alignment of the feature space, thereby alleviating the model bias caused by the difference in the distribution of local multi-modal data and improving the retrieval accuracy. Additionally, we design an adaptive cross-modal knowledge distillation to transfer valuable semantic knowledge from modal-specific global models to local prototype learning processes, reducing the risk of overfitting. Experimental results on three benchmark cross-modal retrieval datasets validate that our PT-FUCH method can achieve outstanding retrieval performance when trained under distributed privacy-preserving mode. The source codes of our method are available at https://github.com/exquisite1210/PT-FUCH_P.

Federated unsupervised cross-modal Hashing

Efficient Discrete Supervised Hashing for Large-scale Cross-modal Retrieval

Semantic Consistency Hashing for Cross-Modal Retrieval

Discrete Similarity Preserving Hashing for Cross-modal Retrieval.

Prototype-guided Knowledge Transfer for Federated Unsupervised Cross-modal Hashing

Supervised Intra- and Inter-Modality Similarity Preserving Hashing for Cross-Modal Retrieval.

High-order nonlocal Hashing for unsupervised cross-modal retrieval

Cross-Domain Transfer Hashing for Efficient Cross-modal Retrieval

Unsupervised Multi-modal Hashing for Cross-Modal Retrieval

Global and Local Semantics-Preserving Based Deep Hashing for Cross-Modal Retrieval

Online supervised collective matrix factorization hashing for cross-modal retrieval

Fusion-Supervised Deep Cross-Modal Hashing

Federated learning for supervised cross-modal retrieval

FedCAFE: Federated Cross-Modal Hashing with Adaptive Feature Enhancement

Deep supervised fused similarity hashing for cross-modal retrieval

A High-Dimensional Sparse Hashing Framework for Cross-Modal Retrieval

Fast Unmediated Hashing for Cross-Modal Retrieval.

Privacy-Enhanced Prototype-based Federated Cross-modal Hashing for Cross-modal Retrieval

Self-supervised incomplete cross-modal hashing retrieval

FedCMR: Federated Cross-Modal Retrieval