Multi-scale Consistency Deep Lifelong Cross-modal Hashing

Liming Xu,Hanqi Li,Jie Shao,Xianhua Zeng,Weisheng Li

DOI: https://doi.org/10.1145/3704636

2024-01-01

Abstract:Deep cross-modal hashing methods provide effective and efficient solutions for large-scale cross-modal retrieval. However, existing cross-modal hashing methods fail to capture the dynamic changes of real-world data, and suffer from serious performance degradation when retrieving streaming data. In this paper, we propose a novel hashing method to achieve accurate cross-modal retrieval under continuous and streaming scenarios. Specifically, regularization-based lifelong learning module is introduced to balance plasticity for learning new knowledge and stability for maintaining old knowledge, and update incremental hash codes without retraining cumulative data. Then, multi-scale consistency network which employs multi-scale feature fusion module to extract fine-grained features among multi-scale modalities is introduced to learn multi-level semantic representations with consistency. Additionally, modality alignment with variational information bottleneck is designed to remove irrelevant information and obtain unified representation, which can be proved to be effective to yield high-quality hash code with new and old knowledge. Extensive experiments show that ours gains the advanced performance and the better adaptability to continuous and streaming environments.