Abstract:Video temporal localization aims to locate a period that semantically matches a natural language query in a given untrimmed video. We empirically observe that although existing approaches gain steady progress on sentence localization, the performance of phrase localization is far from satisfactory. In principle, the phrase should be easier to localize as fewer combinations of visual concepts need to be considered; such incapability indicates that the existing models only capture the sentence annotation bias in the benchmark but lack sufficient understanding of the intrinsic relationship between simple visual and language concepts, thus the model generalization and interpretability is questioned. This paper proposes a unified framework that can deal with both sentence and phrase-level localization, namely Phrase Level Prediction Net (PLPNet). Specifically, based on the hypothesis that similar phrases tend to focus on similar video cues, while dissimilar ones should not, we build a contrastive mechanism to restrain phrase-level localization without fine-grained phrase boundary annotation required in training. Moreover, considering the sentence's flexibility and wide discrepancy among phrases, we propose a clustering-based batch sampler to ensure that contrastive learning can be conducted efficiently. Extensive experiments demonstrate that our method surpasses state-of-the-art methods of phrase-level temporal localization while maintaining high performance in sentence localization and boosting the model's interpretability and generalization capability. Our code is available at https://github.com/sizhelee/PLPNet.

Bilingual Phrase Induction with Local Hard Negative Sampling

Bilingual Lexicon Induction from Non-Parallel Data with Minimal Supervision.

Phrase Localization Without Paired Training Examples

Enhancing Isomorphism Between Word Embedding Spaces for Distant Languages Bilingual Lexicon Induction

Enhancing Bilingual Lexicon Induction Via Bi-directional Translation Pair Retrieving

Improving Bilingual Lexicon Induction on Distant Language Pairs

Enhancing Bilingual Lexicon Induction Via Harnessing Polysemous Words

Discovering Bilingual Lexicons in Polyglot Word Embeddings

Phrase-Level Global-Local Hybrid Model for Sentence Embedding

Phrase-level Prediction for Video Temporal Localization

Bilingual lexicon induction from non-parallel corpora

Bilingual word embedding fusion for robust unsupervised bilingual lexicon induction

Semi-Supervised Learning for Bilingual Lexicon Induction

How Lexical is Bilingual Lexicon Induction?

Unsupervised Bilingual Lexicon Induction Via Latent Variable Models.

Adversarial Training for Unsupervised Bilingual Lexicon Induction

Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

Unsupervised Cross-Lingual Mapping For Phrase Embedding Spaces

Inducing Bilingual Lexica from Non-Parallel Data with Earth Mover's Distance Regularization.

Earth Mover's Distance Minimization for Unsupervised Bilingual Lexicon Induction.

Neighbor Does Matter: Curriculum Global Positive-Negative Sampling for Vision-Language Pre-training