Abstract:Nowadays the amount of multimedia data such as images and text is growing exponentially on social websites, arousing the demand of effective and efficient cross-modal retrieval. The cross-modal hashing based methods have attracted considerable attention recently as they can learn efficient binary codes for heterogeneous data, which enables large-scale similarity search. Generally, to effectively construct the cross-correlation between different modalities, these methods try to find a joint abstraction space where the heterogeneous data can be projected. Then a quantization rule is applied to convert the abstraction representation to binary codes. However, these methods may not effectively bridge the semantic gap through the latent abstraction space because they fail to capture latent information between heterogeneous data. In addition, most of these methods apply the simplest quantization scheme (i.e. sign function) which may cause information loss of the abstraction representation and result in inferior binary codes. To address these challenges, in this paper, we present a novel cross-modal hashing based method that generates unified binary codes combining different modalities. Specifically, we first extract semantic features from the modalities of images and text to capture latent information. Then these semantic features are projected to a joint abstraction space. Finally, the abstraction space is rotated to produce better unified binary codes with much less quantization loss, while preserving the locality structure of projected data. We integrate the binary code learning procedures above to develop an iterative algorithm for optimal solutions. Moreover, we further exploit the useful class label information to reduce the semantic gap between different modalities to benefit the binary code learning. Extensive experiments on four multimedia datasets show that the proposed binary coding schemes outperform several other state-of-the-art methods under cross-modal scenarios.

Unlocking the Potential of Multimodal Unified Discrete Representation Through Training-Free Codebook Optimization and Hierarchical Alignment

Nonlinear Discrete Cross-Modal Hashing for Visual-Textual Data

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Multi-modal Alignment using Representation Codebook

Achieving Cross Modal Generalization with Multimodal Unified Representation.

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

Learning Compact Hash Codes for Multimodal Representations Using Orthogonal Deep Structure.

Multimodal Contrastive Training for Visual Representation Learning

Deep Unified Cross-Modality Hashing by Pairwise Data Alignment

On-the-fly Modulation for Balanced Multimodal Learning

Unifying Discriminative Visual Codebook Generation with Classifier Training for Object Category Recognition

Contrastive Multi-bit Collaborative Learning for Deep Cross-modal Hashing

Unsupervised Dual Hashing Coding (UDC) on Semantic Tagging and Sample Content for Cross-modal Retrieval

Learning unified binary codes for cross-modal retrieval via latent semantic hashing.

Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval

Contrastive Label Correlation Enhanced Unified Hashing Encoder for Cross-modal Retrieval

Deep Cross-Modal Hashing With Hashing Functions and Unified Hash Codes Jointly Learning

Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment

Cross-Modal Retrieval with Discriminative Dual-Path CNN