Abstract:Temporal sentence grounding in videos is a crucial task in vision-language learning. Its goal is retrieving a video segment from an untrimmed video that semantically corresponds to a natural language query. A video usually contains multiple semantic events, which are rarely isolated. They tend to be temporally ordered and semantically correlated (e.g., some event is often the precursor of another event). To precisely localize a semantic moment from a video, it is critical to effectively extract and aggregate multi-granularity contextual information, including the fine-grained local context around the moment-related video segment (in short snippet-level) and coarse-grained semantic correlation (in segment-level). Additionally, a second main insight in this work is that the above context aggregation should be favorably guided by the queries, rather than fully query-agnostic. Putting above ideas together, we here present a new network that does language-guided multi-granularity context aggregation. It is comprised of two major modules. The core of the first module is a novel language-guided temporal adaptive convolution (LTAC) devised to extract fine-grained information over video snippets around the ground-truth video segment. It decomposes a convolution into two channel-oriented / temporal-oriented ones. In particular, the convolutional channels are supposed to be more susceptible to queries, thus we learn to generate a dynamic channel-oriented kernel with respect to the querying sentence. As a second module, we propose a language-guided global relation block (LGRB) that extracts video-level context. It augments the contextual feature by using a multi-scale temporal attention that tackles the scale variation of ground-truth video segments, and a multi-modal semantic attention that relies on syntactic of the query. For the validation purpose, we have conducted comprehensive experiments on two popularly-adopted video benchmarks (i.e., ActivityNet Captions and Charades-STA). All experimental results and ablation studies have clearly corroborated the effectiveness of our model designs, outstripping prior state-of-the-art methods in terms of major performance metrics for the task.

Knowing Where to Leverage: Context-Aware Graph Convolutional Network with an Adaptive Fusion Layer for Contextual Spoken Language Understanding.

Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understanding

Incorporating Complete Syntactical Knowledge for Spoken Language Understanding

CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding

Graph LSTM with Context-Gated Mechanism for Spoken Language Understanding.

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning

GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

Dynamic Time-Aware Attention to Speaker Roles and Contexts for Spoken Language Understanding

A Self-Attention Joint Model for Spoken Language Understanding in Situational Dialog Applications

A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding

End-to-End Cross-Lingual Spoken Language Understanding Model with Multilingual Pretraining.

ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding

Language-Guided Multi-Granularity Context Aggregation for Temporal Sentence Grounding

Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognition

Using Word Confusion Networks for Slot Filling in Spoken Language Understanding.

DiffSLU: Knowledge Distillation Based Diffusion Model for Cross-Lingual Spoken Language Understanding

Using Bidirectional Transformer-CRF for Spoken Language Understanding.

Context-aware Natural Language Generation for Spoken Dialogue Systems.