Abstract:Multi-modal hashing focuses on fusing different modalities and exploring the complementarity of heterogeneous multi-modal data for compact hash learning. However, existing multi-modal hashing methods still suffer from several problems, including: 1) Almost all existing methods generate unexplainable hash codes. They roughly assume that the contribution of each hash code bit to the retrieval results is the same, ignoring the discriminative information embedded in hash learning and semantic similarity in hash retrieval. Moreover, the length of hash code is empirically set, which will cause bit redundancy and affect retrieval accuracy. 2) Most existing methods exploit shallow models which fail to fully capture higher-level correlation of multi-modal data. 3) Most existing methods adopt online hashing strategy based on immutable direct projection, which generates query codes for new samples without considering the differences of semantic categories. In this paper, we propose a Semantic-driven Interpretable Deep Multi-modal Hashing (SIDMH) method to generate interpretable hash codes driven by semantic categories within a deep hashing architecture, which can solve all these three problems in an integrated model. The main contributions are: 1) A novel deep multi-modal hashing network is developed to progressively extract hidden representations of heterogeneous modality features and deeply exploit the complementarity of multi-modal data. 2) Learning interpretable hash codes, with discriminant information of different categories distinctively embedded into hash codes and their different impacts on hash retrieval intuitively explained. Besides, the code length depends on the number of categories in the dataset, which can reduce the bit redundancy and improve the retrieval accuracy. 3) The semantic-driven online hashing strategy encodes the significant branches and discards the negligible branches of each query sample according to the semantics contained in it, therefore it could capture different semantics in dynamic queries. Finally, we consider both the nearest neighbor similarity and semantic similarity of hash codes. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method.

Visual-Semantics Embedding for Deep Hashing-Based Multi-Label Video Retrieval

Scalable Multimedia Retrieval By Deep Learning Hashing With Relative Similarity Learning

Deep Multi-modal Hashing with Semantic Enhancement for Multi-label Micro-video Retrieval

Video Data Hierarchical Retrieval Via Deep Hash Method

Deep Multi-Label Hashing For Large-Scale Visual Search Based On Semantic Graph

Supervised Semantic-Embedded Hashing for Multimedia Retrieval

Unsupervised Deep Video Hashing via Balanced Code for Large-Scale Video Retrieval

Semantic-Driven Interpretable Deep Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Image Retrieval via Balanced and Maximum Variance Deep Hashing.

Attention-Based Video Hashing for Large-Scale Video Retrieval

Classification-enhancement Deep Hashing for Large-Scale Video Retrieval.

Video Retrieval with Similarity-Preserving Deep Temporal Hashing

A Supervised Video Hashing Method Based on a Deep 3D Convolutional Neural Network for Large-Scale Video Retrieval

Deep Visual-Semantic Hashing for Cross-Modal Retrieval

Deep Semantic Multimodal Hashing Network for Scalable Image-Text and Video-Text Retrievals

Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval

Deep and Fast: Deep Learning Hashing with Semi-Supervised Graph Construction.

Attention-based Deep Supervised Hashing for Near Duplicate Video Retrieval

Unsupervised Video Hashing via Deep Neural Network

Video Moment Localization via Deep Cross-Modal Hashing