Abstract:Most image-text retrieval work adopts binary labels indicating whether a pair of image and text matches or not. Such a binary indicator covers only a limited subset of image-text semantic relations, which is insufficient to represent relevance degrees between images and texts described by continuous labels such as image captions. The visual-semantic embedding space obtained by learning binary labels is incoherent and cannot fully characterize the relevance degrees. In addition to the use of binary labels, this paper further incorporates continuous pseudo labels (generally approximated by text similarity between captions) to indicate the relevance degrees. To learn a coherent embedding space, we propose an image-text retrieval framework with Binary and Continuous Label Supervision (BCLS), where binary labels are used to guide the retrieval model to learn limited binary correlations, and continuous labels are complementary to the learning of image-text semantic relations. For the learning of binary labels, we improve the common Triplet ranking loss with Soft Negative mining (Triplet-SN) to improve convergence. For the learning of continuous labels, we design Kendall ranking loss inspired by Kendall rank correlation coefficient (Kendall), which improves the correlation between the similarity scores predicted by the retrieval model and the continuous labels. To mitigate the noise introduced by the continuous pseudo labels, we further design Sliding Window sampling and Hard Sample mining strategy (SW-HS) to alleviate the impact of noise and reduce the complexity of our framework to the same order of magnitude as the triplet ranking loss. Extensive experiments on two image-text retrieval benchmarks demonstrate that our method can improve the performance of state-of-the-art image-text retrieval models.

Exploiting Text Content In Image Search By Semi-Supervised Learning Techniques

Learning A Semantic Space from User'S Relevance Feedback for Image Retrieval

Improving short text classification using public search engines

Enhancing Relevance Feedback in Image Retrieval Using Unlabeled Data

Exploiting image contents in web search

Exploiting Unlabeled Data In Content-Based Image Retrieval

Relevance feedback: A power tool for interactive content-based image retrieval.

Semi-supervised SVM Batch Mode Active Learning for Image Retrieval

Exploiting textual queries for dynamically visual disambiguation

Refining content based image retrieval via semi-supervised learning

Image Retrieval Based on Text and Semantic Relevance Analysis

Image-Text Retrieval with Binary and Continuous Label Supervision

Semisupervised SVM Batch Mode Active Learning with Applications to Image Retrieval

Using Implicit Relevane Feedback to Advance Web Image Search

Self-Adaptive Relevance Feedback Based on Multi-Level Image Content Analysis

Improving Image Retrieval with Semantic Classification Using Relevance Feedback

Search-Based Automatic Web Image Annotation Using Latent Visual and Semantic Analysis

An Image Retrieval Method Based On Collaborative Filtering

Visual and textual fusion for semantically supervised region-based retrieval

Building a Semantic Classification of Image Database from Patterns of Relevance Feedback.

Cross-media relevance mining for evaluating text-based image search engine