Abstract:We propose a framework for parsing video and text jointly for understanding events and answering user queries. Our framework produces a parse graph that represents the compositional structures of spatial information (objects and scenes), temporal information (actions and events) and causal information (causalities between events and fluents) in the video and text. The knowledge representation of our framework is based on a spatial-temporal-causal And-Or graph (S/T/C-AOG), which jointly models possible hierarchical compositions of objects, scenes and events as well as their interactions and mutual contexts, and specifies the prior probabilistic distribution of the parse graphs. We present a probabilistic generative model for joint parsing that captures the relations between the input video/text, their corresponding parse graphs and the joint parse graph. Based on the probabilistic model, we propose a joint parsing system consisting of three modules: video parsing, text parsing and joint inference. Video parsing and text parsing produce two parse graphs from the input video and text respectively. The joint inference module produces a joint parse graph by performing matching, deduction and revision on the video and text parse graphs. The proposed framework has the following objectives: Firstly, we aim at deep semantic parsing of video and text that goes beyond the traditional bag-of-words approaches; Secondly, we perform parsing and reasoning across the spatial, temporal and causal dimensions based on the joint S/T/C-AOG representation; Thirdly, we show that deep joint parsing facilitates subsequent applications such as generating narrative text descriptions and answering queries in the forms of who, what, when, where and why. We empirically evaluated our system based on comparison against ground-truth as well as accuracy of query answering and obtained satisfactory results.

Joint Multi-Decoder Framework with Hierarchical Pointer Network for Frame Semantic Parsing.

A Double-Graph Based Framework for Frame Semantic Parsing

Local-to-Global Framework for Multi-target Frame Identification.

A Span-based Target-aware Relation Model for Frame-semantic Parsing.

Hierarchical Pointer Net Parsing

Non-rigid Video Object Segmentation Based on Semantic Multi-level Framework.

Multi-Class Part Parsing with Joint Boundary-Semantic Awareness.

Joint Semantic Understanding with a Multilevel Branch for Driving Perception

Hierarchical Human Semantic Parsing With Comprehensive Part-Relation Modeling

Knowledge-augmented Frame Semantic Parsing with Hybrid Prompt-tuning

Boosting Real-Time Driving Scene Parsing with Shared Semantics

Deep Hierarchical Human Semantic Parsing

Multi-Layer Pseudo-Siamese Biaffine Model for Dependency Parsing.

Here's My Point: Joint Pointer Architecture for Argument Mining

A Representation Learning Framework For Multi-Source Transfer Parsing

A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling

Parsing All: Syntax and Semantics, Dependencies and Spans

High-order Joint Constituency and Dependency Parsing

Hierarchical Human Parsing with Typed Part-Relation Reasoning

Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing

Joint Video and Text Parsing for Understanding Events and Answering Queries