Abstract:Deep cross-modal hashing has promoted the field of multi-modal retrieval due to its excellent efficiency and storage, but its vulnerability to backdoor attacks is rarely studied. Notably, current deep cross-modal hashing methods inevitably require large-scale training data, resulting in poisoned samples with imperceptible triggers that can easily be camouflaged into the training data to bury backdoors in the victim model. Nevertheless, existing backdoor attacks focus on the uni-modal vision domain, while the multi-modal gap and hash quantization weaken their attack performance. In addressing the aforementioned challenges, we undertake an invisible black-box backdoor attack against deep cross-modal hashing retrieval in this paper. To the best of our knowledge, this is the first attempt in this research field. Specifically, we develop a flexible trigger generator to generate the attacker’s specified triggers, which learns the sample semantics of the non-poisoned modality to bridge the cross-modal attack gap. Then, we devise an input-aware injection network, which embeds the generated triggers into benign samples in the form of sample-specific stealth and realizes cross-modal semantic interaction between triggers and poisoned samples. Owing to the knowledge-agnostic of victim models, we enable any cross-modal hashing knockoff to facilitate the black-box backdoor attack and alleviate the attack weakening of hash quantization. Moreover, we propose a confusing perturbation and mask strategy to induce the high-performance victim models to focus on imperceptible triggers in poisoned samples. Extensive experiments on benchmark datasets demonstrate that our method has a state-of-the-art attack performance against deep cross-modal hashing retrieval. Besides, we investigate the influences of transferable attacks, few-shot poisoning, multi-modal poisoning, perceptibility, and potential defenses on backdoor attacks. Our codes and datasets are available at https://github.com/tswang0116/IB3A.

I Can Retrieve More Than Images: Contrastive Stealing Attack Against Deep Hashing Models

You See What I Want You to See: Exploring Targeted Black-Box Transferability Attack for Hash-based Image Retrieval Systems

Evade Deep Image Retrieval by Stashing Private Images in the Hash Space.

The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks

A GAN-Based Defense Framework Against Model Inversion Attacks.

Towards Efficient Data Free Blackbox Adversarial Attack

Efficient Non-Targeted Attack for Deep Hashing Based Image Retrieval

Targeted Attack of Deep Hashing via Prototype-supervised Adversarial Networks

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

Clean-label Backdoor Attack Against Deep Hashing Based Retrieval.

Semantic-Aware Adversarial Training for Reliable Deep Hashing Retrieval

Reliable and Efficient Evaluation of Adversarial Robustness for Deep Hashing-Based Retrieval

CoopHash: Cooperative Learning of Multipurpose Descriptor and Contrastive Pair Generator via Variational MCMC Teaching for Supervised Image Hashing

Invisible Black-Box Backdoor Attack against Deep Cross-Modal Hashing Retrieval

HashGAN: Deep Learning to Hash with Pair Conditional Wasserstein GAN

Targeted Adversarial Attack against Deep Cross-modal Hashing Retrieval

Targeted Attack for Deep Hashing based Retrieval

Deep Self-Adaptive Hashing for Image Retrieval

Deep Progressive Hashing for Image Retrieval

Backdoor Attack on Hash-based Image Retrieval via Clean-label Data Poisoning.