Abstract:This paper studies the multimedia problem of temporal sentence grounding (TSG), which aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query. Traditional TSG methods mainly follow the top-down or bottom-up framework and are not end-to-end. They severely rely on time-consuming post-processing to refine the grounding results. Recently, some transformer-based approaches are proposed to efficiently and effectively model the fine-grained semantic alignment between video and query. Although these methods achieve significant performance to some extent, they equally take frames of the video and words of the query as transformer input for correlating, failing to capture their different levels of granularity with distinct semantics. To address this issue, in this paper, we propose a novel Hierarchical Local-Global Transformer (HLGT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities for learning more fine-grained multi-modal representations. Specifically, we first split the video and query into individual clips and phrases to learn their local context (adjacent dependency) and global correlation (long-range dependency) via a temporal transformer. Then, a global-local transformer is introduced to learn the interactions between the local-level and global-level semantics for better multi-modal reasoning. Besides, we develop a new cross-modal cycle-consistency loss to enforce interaction between two modalities and encourage the semantic alignment between them. Finally, we design a brand-new cross-modal parallel transformer decoder to integrate the encoded visual and textual features for final grounding. Extensive experiments on three challenging datasets show that our proposed HLGT achieves a new state-of-the-art performance.

Local and Global Contexts for Conversation

Local and Global Context Modeling with Relation Matching Task for Dialog Act Recognition

Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation

An Empirical Study on Context Length for Open-Domain Dialog Generation

Extending the Transformer with Context and Multi-dimensional Mechanism for Dialogue Response Generation.

Knowing Where to Leverage: Context-Aware Graph Convolutional Network with an Adaptive Fusion Layer for Contextual Spoken Language Understanding.

One Type Context is Not Enough: Global Context-Aware Neural Machine Translation

Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension

Context-LGM: Leveraging Object-Context Relation for Context-Aware Object Recognition

Local Topology Measures of Contextual Language Model Latent Spaces With Applications to Dialogue Term Extraction

Towards Making the Most of Context in Neural Machine Translation

Dialogue emotion model based on local–global context encoder and commonsense knowledge fusion attention

Improving BERT with local context comprehension for multi-turn response selection in retrieval-based dialogue systems

GNN-LM: Language Modeling based on Global Contexts via GNN

Inferring Speaking Styles from Multi-modal Conversational Context by Multi-scale Relational Graph Convolutional Networks

Empower Your Model with Longer and Better Context Comprehension

Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition

Hierarchical Local-Global Transformer for Temporal Sentence Grounding.

Contextualized Non-Local Neural Networks for Sequence Learning

Improving Matching Models with Hierarchical Contextualized Representations for Multi-turn Response Selection

Channel-Aware Decoupling Network for Multiturn Dialog Comprehension.