Abstract:Nowadays the amount of multimedia data such as images and text is growing exponentially on social websites, arousing the demand of effective and efficient cross-modal retrieval. The cross-modal hashing based methods have attracted considerable attention recently as they can learn efficient binary codes for heterogeneous data, which enables large-scale similarity search. Generally, to effectively construct the cross-correlation between different modalities, these methods try to find a joint abstraction space where the heterogeneous data can be projected. Then a quantization rule is applied to convert the abstraction representation to binary codes. However, these methods may not effectively bridge the semantic gap through the latent abstraction space because they fail to capture latent information between heterogeneous data. In addition, most of these methods apply the simplest quantization scheme (i.e. sign function) which may cause information loss of the abstraction representation and result in inferior binary codes. To address these challenges, in this paper, we present a novel cross-modal hashing based method that generates unified binary codes combining different modalities. Specifically, we first extract semantic features from the modalities of images and text to capture latent information. Then these semantic features are projected to a joint abstraction space. Finally, the abstraction space is rotated to produce better unified binary codes with much less quantization loss, while preserving the locality structure of projected data. We integrate the binary code learning procedures above to develop an iterative algorithm for optimal solutions. Moreover, we further exploit the useful class label information to reduce the semantic gap between different modalities to benefit the binary code learning. Extensive experiments on four multimedia datasets show that the proposed binary coding schemes outperform several other state-of-the-art methods under cross-modal scenarios.

Semantics-Aware Spatial-Temporal Binaries for Cross-Modal Video Retrieval

Semantic Consistency Hashing for Cross-Modal Retrieval

Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval

Online Cross-Modal Scene Retrieval by Binary Representation and Semantic Graph

Learning unified binary codes for cross-modal retrieval via latent semantic hashing.

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Semantic Binary Coding for Visual Recognition Via Joint Concept-Attribute Modelling.

Efficient Binary Coding for Subspace-based Query-by-Image Video Retrieval

Based on Spatial and Temporal Implicit Semantic Relational Inference for Cross-Modal Retrieval

Hybrid Space Learning for Language-based Video Retrieval

Spatial-temporal Graphs for Cross-modal Text2Video Retrieval

Boosting Temporal Binary Coding for Large-Scale Video Search

Dual Encoding for Video Retrieval by Text

Video Retrieval with Similarity-Preserving Deep Temporal Hashing

Binary Subspace Coding for Query-by-Image Video Retrieval

Scalable Video Event Retrieval by Visual State Binary Embedding.

Binary Feature Representation Learning for Scene Retrieval in Micro-Video

Multi-grained Encoding and Joint Embedding Space Fusion for Video and Text Cross-Modal Retrieval

Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval

Coupled Binary Embedding for Large-Scale Image Retrieval

Exploiting Concept Correlation with Attributes for Semantic Binary Representation Learning.