Abstract:As multi-modal data proliferates, people are no longer content with a single mode of data retrieval for access to information. Deep hashing retrieval algorithms have attracted much attention for their advantages of efficient storage and fast query speed. Currently, the existing unsupervised hashing methods generally have two limitations: (1) Existing methods fail to adequately capture the latent semantic relevance and coexistent information from the different modality data, resulting in the lack of effective feature and hash encoding representation to bridge the heterogeneous and semantic gaps in multi-modal data. (2) Existing unsupervised methods typically construct a similarity matrix to guide the hash code learning, which suffers from inaccurate similarity problems, resulting in sub-optimal retrieval performance. To address these issues, we propose a novel CLIP-based fusion-modal reconstructing hashing for Large-scale Unsupervised Cross-modal Retrieval. First, we use CLIP to encode cross-modal features of visual modalities, and learn the common representation space of the hash code using modality-specific autoencoders. Second, we propose an efficient fusion approach to construct a semantically complementary affinity matrix that can maximize the potential semantic relevance of different modal instances. Furthermore, to retain the intrinsic semantic similarity of all similar pairs in the learned hash codes, an objective function for similarity reconstruction based on semantic complementation is designed to learn high-quality hash code representations. Sufficient experiments were carried out on four multi-modal benchmark datasets (WIKI, MIRFLICKR, NUS-WIDE, and MS COCO), and the proposed method achieves state-of-the-art image-text retrieval performance compared to several representative unsupervised cross-modal hashing methods.

Semantic-rebased cross-modal hashing for scalable unsupervised text-visual retrieval

Semantic Consistency Hashing for Cross-Modal Retrieval

Efficient Discrete Supervised Hashing for Large-scale Cross-modal Retrieval

Supervised Coarse-to-Fine Semantic Hashing for Cross-Media Retrieval.

Nonlinear Discrete Cross-Modal Hashing for Visual-Textual Data

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Set and Rebase: Determining the Semantic Graph Connectivity for Unsupervised Cross-Modal Hashing

Semantic-consistent cross-modal hashing for large-scale image retrieval

Semi-supervised Semi-paired Cross-modal Hashing

Semantic embedding based online cross-modal hashing method

Scalable Unsupervised Hashing via Exploiting Robust Cross-modal Consistency

A High-Dimensional Sparse Hashing Framework for Cross-Modal Retrieval

Pseudo-label driven deep hashing for unsupervised cross-modal retrieval

Deep Cross-modal Hashing Based on Semantic Consistent Ranking

Unsupervised Cross-modal Hashing via Semantic Text Mining

Deep noise mitigation and semantic reconstruction hashing for unsupervised cross-modal retrieval

CLIP-based fusion-modal reconstructing hashing for large-scale unsupervised cross-modal retrieval

Unsupervised Multi-modal Hashing for Cross-Modal Retrieval

Deep Multi-Level Semantic Hashing for Cross-Modal Retrieval

Deep Semantic-Alignment Hashing for Unsupervised Cross-Modal Retrieval