Abstract:Although deep cross-modal hashing methods have shown superiorities for cross-modal retrieval recently, there is a concern about potential data privacy leakage when training the models. Federated learning adopts a distributed machine learning strategy, which can collaboratively train models without leaking local private data. It is a promising technique to support privacy-preserving cross-modal hashing. However, existing federated learning-based cross-modal retrieval methods usually rely on a large number of semantic annotations, which limits the scalability of the retrieval models. Furthermore, they mostly update the global models by aggregating local model parameters, ignoring the differences in the quantity and category of multi-modal data from multiple clients. To address these issues, we propose a Prototype Transfer-based Federated Unsupervised Cross-modal Hashing(PT-FUCH) method for solving the privacy leakage problem in cross-modal retrieval model learning. PT-FUCH protects local private data by exploring unified global prototypes for different clients, without relying on any semantic annotations. Global prototypes are used to guide the local cross-modal hash learning and promote the alignment of the feature space, thereby alleviating the model bias caused by the difference in the distribution of local multi-modal data and improving the retrieval accuracy. Additionally, we design an adaptive cross-modal knowledge distillation to transfer valuable semantic knowledge from modal-specific global models to local prototype learning processes, reducing the risk of overfitting. Experimental results on three benchmark cross-modal retrieval datasets validate that our PT-FUCH method can achieve outstanding retrieval performance when trained under distributed privacy-preserving mode. The source codes of our method are available at https://github.com/exquisite1210/PT-FUCH_P.

Privacy-Enhanced Prototype-based Federated Cross-modal Hashing for Cross-modal Retrieval

Discrete Similarity Preserving Hashing for Cross-modal Retrieval.

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Prototype-guided Knowledge Transfer for Federated Unsupervised Cross-modal Hashing

Prototype-Based Layered Federated Cross-Modal Hashing

Federated unsupervised cross-modal Hashing

Online Cross-modal Hashing With Dynamic Prototype

FedHAP: Federated Hashing with Global Prototypes for Cross-silo Retrieval

FedCAFE: Federated Cross-Modal Hashing with Adaptive Feature Enhancement

Fast Unmediated Hashing for Cross-Modal Retrieval.

Two-step Heterogeneous Fusion Hashing Network for Unpaired Cross-Modal Retrieval

DAP2CMH: Deep Adversarial Privacy-Preserving Cross-Modal Hashing

Fast Unsupervised Consistent and Modality-Specific Hashing for Multimedia Retrieval.

Data-Focus Proxy Hashing

Discrete Fusion Adversarial Hashing for Cross-Modal Retrieval

MTFH: A Matrix Tri-Factorization Hashing Framework for Efficient Cross-Modal Retrieval

Cross-Domain Transfer Hashing for Efficient Cross-modal Retrieval

Two-Stage Asymmetric Similarity Preserving Hashing for Cross-Modal Retrieval

Proxy-based Graph Convolutional Hashing for Cross-Modal Retrieval

CLIP-HASH: A Lightweight Hashing Network for Cross-Modal Retrieval.

SCH: Symmetric Consistent Hashing for Cross-Modal Retrieval