Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

liyang he,Zhenya Huang,Jiayu Liu,Enhong Chen,Fei Wang,Jing Sha,Shijin Wang
DOI: https://doi.org/10.1145/3589334.3645440
2024-01-01
Abstract:Unsupervised semantic hashing has emerged as an indispensable technique forfast image search, which aims to convert images into binary hash codes withoutrelying on labels. Recent advancements in the field demonstrate that employinglarge-scale backbones (e.g., ViT) in unsupervised semantic hashing models canyield substantial improvements. However, the inference delay has becomeincreasingly difficult to overlook. Knowledge distillation provides a means forpractical model compression to alleviate this delay. Nevertheless, theprevailing knowledge distillation approaches are not explicitly designed forsemantic hashing. They ignore the unique search paradigm of semantic hashing,the inherent necessities of the distillation process, and the property of hashcodes. In this paper, we propose an innovative Bit-mask Robust Contrastiveknowledge Distillation (BRCD) method, specifically devised for the distillationof semantic hashing models. To ensure the effectiveness of two kinds of searchparadigms in the context of semantic hashing, BRCD first aligns the semanticspaces between the teacher and student models through a contrastive knowledgedistillation objective. Additionally, to eliminate noisy augmentations andensure robust optimization, a cluster-based method within the knowledgedistillation process is introduced. Furthermore, through a bit-level analysis,we uncover the presence of redundancy bits resulting from the bit independenceproperty. To mitigate these effects, we introduce a bit mask mechanism in ourknowledge distillation objective. Finally, extensive experiments not onlyshowcase the noteworthy performance of our BRCD method in comparison to otherknowledge distillation methods but also substantiate the generality of ourmethods across diverse semantic hashing models and backbones. The code for BRCDis available at https://github.com/hly1998/BRCD.
What problem does this paper attempt to address?