Abstract:Although deep cross-modal hashing methods have shown superiorities for cross-modal retrieval recently, there is a concern about potential data privacy leakage when training the models. Federated learning adopts a distributed machine learning strategy, which can collaboratively train models without leaking local private data. It is a promising technique to support privacy-preserving cross-modal hashing. However, existing federated learning-based cross-modal retrieval methods usually rely on a large number of semantic annotations, which limits the scalability of the retrieval models. Furthermore, they mostly update the global models by aggregating local model parameters, ignoring the differences in the quantity and category of multi-modal data from multiple clients. To address these issues, we propose a Prototype Transfer-based Federated Unsupervised Cross-modal Hashing(PT-FUCH) method for solving the privacy leakage problem in cross-modal retrieval model learning. PT-FUCH protects local private data by exploring unified global prototypes for different clients, without relying on any semantic annotations. Global prototypes are used to guide the local cross-modal hash learning and promote the alignment of the feature space, thereby alleviating the model bias caused by the difference in the distribution of local multi-modal data and improving the retrieval accuracy. Additionally, we design an adaptive cross-modal knowledge distillation to transfer valuable semantic knowledge from modal-specific global models to local prototype learning processes, reducing the risk of overfitting. Experimental results on three benchmark cross-modal retrieval datasets validate that our PT-FUCH method can achieve outstanding retrieval performance when trained under distributed privacy-preserving mode. The source codes of our method are available at https://github.com/exquisite1210/PT-FUCH_P.

FedHAP: Federated Hashing with Global Prototypes for Cross-silo Retrieval

Efficient Discrete Supervised Hashing for Large-scale Cross-modal Retrieval

Prototype-Based Layered Federated Cross-Modal Hashing

FedDGP: Disentangling Global and Personal Models for Federated Learning

Federated Patient Hashing.

Deep Transfer Hashing for Adaptive Learning on Federated Streaming Data

Privacy-Enhanced Prototype-based Federated Cross-modal Hashing for Cross-modal Retrieval

Prototype-guided Knowledge Transfer for Federated Unsupervised Cross-modal Hashing

Dynamic Heterogeneous Federated Learning with Multi-Level Prototypes

Federating Hashing Networks Adaptively for Privacy-Preserving Retrieval

Federated unsupervised cross-modal Hashing

FedCAFE: Federated Cross-Modal Hashing with Adaptive Feature Enhancement

Deep Hierarchy-aware Proxy Hashing with Self-paced Learning for Cross-modal Retrieval

HyperFed: Hyperbolic Prototypes Exploration with Consistent Aggregation for Non-IID Data in Federated Learning

Global and Local Semantics-Preserving Based Deep Hashing for Cross-Modal Retrieval

Deep Neighborhood-aware Proxy Hashing with Uniform Distribution Constraint for Cross-modal Retrieval

Discrete Fusion Adversarial Hashing for Cross-Modal Retrieval

Federated Multiple Label Hashing (FedMLH): Communication Efficient Federated Learning on Extreme Classification Tasks

Learning discrete class-specific prototypes for deep semantic hashing

Fusion-Supervised Deep Cross-Modal Hashing