Semantic embedding based online cross-modal hashing method

Meijia Zhang,Junzheng Li,Xiyuan Zheng

DOI: https://doi.org/10.1038/s41598-023-50242-w

IF: 4.6

2024-01-07

Scientific Reports

Abstract:Hashing has been extensively utilized in cross-modal retrieval due to its high efficiency in handling large-scale, high-dimensional data. However, most existing cross-modal hashing methods operate as offline learning models, which learn hash codes in a batch-based manner and prove to be inefficient for streaming data. Recently, several online cross-modal hashing methods have been proposed to address the streaming data scenario. Nevertheless, these methods fail to fully leverage the semantic information and accurately optimize hashing in a discrete fashion. As a result, both the accuracy and efficiency of online cross-modal hashing methods are not ideal. To address these issues, this paper introduces the Semantic Embedding-based Online Cross-modal Hashing (SEOCH) method, which integrates semantic information exploitation and online learning into a unified framework. To exploit the semantic information, we map the semantic labels to a latent semantic space and construct a semantic similarity matrix to preserve the similarity between new data and existing data in the Hamming space. Moreover, we employ a discrete optimization strategy to enhance the efficiency of cross-modal retrieval for online hashing. Through extensive experiments on two publicly available multi-label datasets, we demonstrate the superiority of the SEOCH method.

multidisciplinary sciences

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily addresses the issues of efficiency and accuracy in cross-modal retrieval in streaming data scenarios. Specifically:

1. **Problems with Existing Methods**:
- Most existing cross-modal hashing methods use offline learning models, learning hash codes in a batch manner, which is inefficient when dealing with large-scale, high-dimensional data.
- Some recently proposed online cross-modal hashing methods address the streaming data problem but fail to fully utilize semantic information and do not accurately optimize discrete hashing.

2. **Proposed Solution**:
- An online cross-modal hashing method based on semantic embedding (SEOCH) is proposed, integrating semantic information utilization and online learning into a unified framework.
- By mapping semantic labels to a latent semantic space and constructing a semantic similarity matrix, the similarity between new data and existing data is maintained.
- A discrete optimization strategy is adopted to improve the efficiency of online hashing.

3. **Experimental Validation**:
- Extensive experiments were conducted on two public multi-label datasets (MIRFLICKR-25K and NUS-WIDE), demonstrating the effectiveness and superiority of the SEOCH method.

Through the above methods, the paper aims to enhance the accuracy and efficiency of cross-modal retrieval in streaming data scenarios.