Abstract:This paper studies the multimedia problem of temporal sentence grounding (TSG), which aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query. Traditional TSG methods mainly follow the top-down or bottom-up framework and are not end-to-end. They severely rely on time-consuming post-processing to refine the grounding results. Recently, some transformer-based approaches are proposed to efficiently and effectively model the fine-grained semantic alignment between video and query. Although these methods achieve significant performance to some extent, they equally take frames of the video and words of the query as transformer input for correlating, failing to capture their different levels of granularity with distinct semantics. To address this issue, in this paper, we propose a novel Hierarchical Local-Global Transformer (HLGT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities for learning more fine-grained multi-modal representations. Specifically, we first split the video and query into individual clips and phrases to learn their local context (adjacent dependency) and global correlation (long-range dependency) via a temporal transformer. Then, a global-local transformer is introduced to learn the interactions between the local-level and global-level semantics for better multi-modal reasoning. Besides, we develop a new cross-modal cycle-consistency loss to enforce interaction between two modalities and encourage the semantic alignment between them. Finally, we design a brand-new cross-modal parallel transformer decoder to integrate the encoded visual and textual features for final grounding. Extensive experiments on three challenging datasets show that our proposed HLGT achieves a new state-of-the-art performance.

Compositional Video Understanding with Spatiotemporal Structure-based Transformers

TransVOS: Video Object Segmentation with Transformers

Learning Spatiotemporal Relationships with a Unified Framework for Video Object Segmentation

Towards Long-Form Video Understanding

Time Is MattEr: Temporal Self-supervision for Video Transformers

Transformer-based Video Saliency Prediction with High Temporal Dimension Decoding

Understanding Video Transformers via Universal Concept Discovery

Hierarchical Spatiotemporal Transformers for Video Object Segmentation

RESTHT: relation-enhanced spatial–temporal hierarchical transformer for video captioning

Deep Transformer Based Video Inpainting Using Fast Fourier Tokenization

Hierarchical Separable Video Transformer for Snapshot Compressive Imaging

Exploiting long-term temporal dynamics for video captioning

A Neural ODE and Transformer-based Model for Temporal Understanding and Dense Video Captioning

TAM-VT: Transformation-Aware Multi-scale Video Transformer for Segmentation and Tracking

Multiview Transformers for Video Recognition

Hierarchical Local-Global Transformer for Temporal Sentence Grounding.

Convolutional Tensor-Train LSTM for Spatio-temporal Learning

Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation

Two-Stream Transformer Architecture for Long Video Understanding

Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning

A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection