Abstract:To retrieve a video via a multimedia search engine, a textual query is usually created by the user and then used to perform the search. Recent state-of-the-art cross-modal retrieval methods learn a joint text-video embedding space by using contrastive loss functions, which maximize the similarity of positive pairs while decreasing that of the negative pairs. Although the choice of these pairs is fundamental for the construction of the joint embedding space, the selection procedure is usually driven by the relationships found within the dataset: a positive pair is commonly formed by a video and its own caption, whereas unrelated video-caption pairs represent the negative ones. We hypothesize that this choice results in a retrieval system with limited semantics understanding, as the standard training procedure requires the system to discriminate between groundtruth and negative even though there is no difference in their semantics. Therefore, differently from the previous approaches, in this paper we propose a novel strategy for the selection of both positive and negative pairs which takes into account both the annotations and the semantic contents of the captions. By doing so, the selected negatives do not share semantic concepts with the positive pair anymore, and it is also possible to discover new positives within the dataset. Based on our hypothesis, we provide a novel design of two popular contrastive loss functions, and explore their effectiveness on three heterogeneous state-of-the-art approaches. The extensive experimental analysis conducted on two datasets, EPIC-Kitchens-100 and MSR-VTT, validates the effectiveness of the proposed strategy, observing, e.g., more than +20% nDCG on EPIC-Kitchens-100. Furthermore, these results are corroborated with qualitative evidence both supporting our hypothesis and explaining why the proposed strategy effectively overcomes it.

Semantic video indexing by fusing explicit and implicit context spaces.

Exploring Inter-Concept Relationship with Context Space for Semantic Video Indexing

Global and Compact Video Context Embedding for Video Semantic Segmentation

Semantic Context Transfer Across Heterogeneous Sources for Domain Adaptive Video Search.

Video diver: generic video indexing with diverse features.

Concept-Driven Multi-Modality Fusion for Video Search

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Video search in concept subspace: a text-like paradigm

Efficient Heuristic Methods for Multimodal Fusion and Concept Fusion in Video Concept Detection

Robust Semantic Concept Detection in Large Video Collections

Semantic Video Search by Exploiting Large-Scale Visual Concepts

Graph-based Multi-Space Semantic Correlation Propagation for Video Retrieval

Exploiting Semantic And Visual Context For Effective Video Annotation

Research of improving semantic image segmentation based on a feature fusion model

Based on Spatial and Temporal Implicit Semantic Relational Inference for Cross-Modal Retrieval

Exploiting Concept Association to Boost Multimedia Semantic Concept Detection

Mapping Query to Semantic Concepts: Leveraging Semantic Indices for Automatic and Interactive Video Retrieval

Video Semantic Concept Detection Using Multi-Modality Subspace Correlation Propagation

Explicit and implicit concept-based video retrieval with bipartite graph propagation model.

Improving semantic video retrieval models by training with a relevance-aware online mining strategy

Interpretable Embedding for Ad-hoc Video Search