Abstract:Nowadays the amount of multimedia data such as images and text is growing exponentially on social websites, arousing the demand of effective and efficient cross-modal retrieval. The cross-modal hashing based methods have attracted considerable attention recently as they can learn efficient binary codes for heterogeneous data, which enables large-scale similarity search. Generally, to effectively construct the cross-correlation between different modalities, these methods try to find a joint abstraction space where the heterogeneous data can be projected. Then a quantization rule is applied to convert the abstraction representation to binary codes. However, these methods may not effectively bridge the semantic gap through the latent abstraction space because they fail to capture latent information between heterogeneous data. In addition, most of these methods apply the simplest quantization scheme (i.e. sign function) which may cause information loss of the abstraction representation and result in inferior binary codes. To address these challenges, in this paper, we present a novel cross-modal hashing based method that generates unified binary codes combining different modalities. Specifically, we first extract semantic features from the modalities of images and text to capture latent information. Then these semantic features are projected to a joint abstraction space. Finally, the abstraction space is rotated to produce better unified binary codes with much less quantization loss, while preserving the locality structure of projected data. We integrate the binary code learning procedures above to develop an iterative algorithm for optimal solutions. Moreover, we further exploit the useful class label information to reduce the semantic gap between different modalities to benefit the binary code learning. Extensive experiments on four multimedia datasets show that the proposed binary coding schemes outperform several other state-of-the-art methods under cross-modal scenarios.

Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Nonlinear Discrete Cross-Modal Hashing for Visual-Textual Data

Semantic Consistency Hashing for Cross-Modal Retrieval

Semantics-Aware Spatial-Temporal Binaries for Cross-Modal Video Retrieval

Deep Visual-Semantic Hashing for Cross-Modal Retrieval

Deep Binary Reconstruction for Cross-modal Hashing

Stacked Convolutional Deep Encoding Network for Video-Text Retrieval.

Learning unified binary codes for cross-modal retrieval via latent semantic hashing.

Discriminative Cross-View Binary Representation Learning

Deep Voice-Visual Cross-Modal Retrieval with Deep Feature Similarity Learning

Scalable Video Event Retrieval by Visual State Binary Embedding.

Deep Variational and Structural Hashing.

Efficient Cross-Modal Retrieval via Deep Binary Hashing and Quantization

Dual Variational Network for Unsupervised Cross-Modal Hashing

Deep Binary Representation for Efficient Image Retrieval.

Multimodal Discriminative Binary Embedding for Large-Scale Cross-Modal Retrieval.

Video Retrieval with Similarity-Preserving Deep Temporal Hashing

Learning Discriminative Binary Codes for Large-scale Cross-modal Retrieval

Deep Supervised Hashing for Fast Image Retrieval

Coupled Binary Embedding for Large-Scale Image Retrieval