Unsupervised semantic hashing has emerged as an indispensable technique for fast image search, which aims to convert images into binary hash codes without relying on labels. Recent advancements in the field demonstrate that employing large-scale backbones (e.g., ViT) in unsupervised semantic hashing models can yield substantial improvements. However, the inference delay has become increasingly difficult to overlook. Knowledge distillation provides a means for practical model compression to alleviate this delay. Nevertheless, the prevailing knowledge distillation approaches are not explicitly designed for semantic hashing. They ignore the unique search paradigm of semantic hashing, the inherent necessities of the distillation process, and the property of hash codes. In this paper, we propose an innovative Bit-mask Robust Contrastive knowledge Distillation (BRCD) method, specifically devised for the distillation of semantic hashing models. To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective. Additionally, to eliminate noisy augmentations and ensure robust optimization, a cluster-based method within the knowledge distillation process is introduced. Furthermore, through a bit-level analysis, we uncover the presence of redundancy bits resulting from the bit independence property. To mitigate these effects, we introduce a bit mask mechanism in our knowledge distillation objective. Finally, extensive experiments not only showcase the noteworthy performance of our BRCD method in comparison to other knowledge distillation methods but also substantiate the generality of our methods across diverse semantic hashing models and backbones. The code for BRCD is available at https://github.com/hly1998/BRCD.

本文提出了一种创新的位掩码鲁棒对比知识蒸馏（BRCD）方法，专门用于蒸馏语义哈希模型，在语义哈希的上下文中，通过对比知识蒸馏目标，首先使教师和学生模型之间的语义空间相互对齐。此外，在知识蒸馏过程中引入了基于聚类的方法，以消除噪声增强，并确保鲁棒优化。通过位级分析，我们发现了由于位独立性属性而产生的冗余位的存在。为了减轻这些影响，我们在知识蒸馏目标中引入了位掩码机制。最后，广泛的实验不仅展示了我们的BRCD方法相对于其他知识蒸馏方法的显着性能，还证实了我们的方法在不同的语义哈希模型和骨干网络中的普遍性。

位掩码鲁棒对比式知识蒸馏用于无监督语义哈希