Abstract:Video-text retrieval is an important yet challenging task in visionlanguage understanding, which aims to learn a joint embedding space where related video and text instances are close to each other. Most current works simply measure the video-text similarity based on video-level and text-level embeddings. However, the neglect of more fine-grained or local information causes the problem of insufficient representation. Some works exploit the local details by disentangling sentences, but overlook the corresponding videos, causing the asymmetry of video-text representation. To address the above limitations, we propose a Hierarchical Alignment Network (HANet) to align different level representations for video-text matching. Specifically, we first decompose video and text into three semantic levels, namely event (video and text), action (motion and verb), and entity (appearance and noun). Based on these, we naturally construct hierarchical representations in the individual-localglobal manner, where the individual level focuses on the alignment between frame and word, local level focuses on the alignment between video clip and textual context, and global level focuses on the alignment between the whole video and text. Different level alignments capture fine-to-coarse correlations between video and text, as well as take the advantage of the complementary information among three semantic levels. Besides, our HANet is also richly interpretable by explicitly learning key semantic concepts. Extensive experiments on two public datasets, namely MSR-VTT and VATEX, show the proposed HANet outperforms other state-of-theart methods, which demonstrates the effectiveness of hierarchical representation and alignment. Our code is publicly available at https://github.com/Roc-Ng/HANet.

Hierarchically Browse and Annotation System for News Video

Description and Browsing of Video Story Structure

Hierarchical organization for medical video summarization using latent visual and semantic analysis

Hierarchical Video Data Modeling and Indexing for Virtual Scene Construction

A New Method for Text Location in News Video Based on Ant Colony Algorithm

Video News Indexing Using Semantic-Face

Annotating and navigating tourist videos.

Research on Anchorperson Detection Method in News Video

Video abstraction based on the visual attention model and online clustering

A Method to Detect Anchorperson Shots for Digital TV News

News Video Indexing and Abstraction by Specific Visual Cues

Configurable Spatial-Temporal Hierarchical Analysis for Flexible Video Anomaly Detection

Automatic Segmentation of News Items Based on Video and Audio Features

Video Abstraction via Attention Model and On-Line Clustering

Multiple style exploration for story unit segmentation of broadcast news video

A News Video Mining Method Based On Statistical Analysis And Visualization

HANet: Hierarchical Alignment Networks for Video-Text Retrieval

A Scalable Video Search Engine Based on Audio Content Indexing and Topic Segmentation

Hierarchical Visualization of Video Search Results for Topic-Based Browsing.

InsightVideo: toward hierarchical video content organization for efficient browsing, summarization and retrieval

A New Method of News Local-caption Extraction Based on Spatio-temporal Distribution Feature