Abstract:We propose a framework for parsing video and text jointly for understanding events and answering user queries. Our framework produces a parse graph that represents the compositional structures of spatial information (objects and scenes), temporal information (actions and events) and causal information (causalities between events and fluents) in the video and text. The knowledge representation of our framework is based on a spatial-temporal-causal And-Or graph (S/T/C-AOG), which jointly models possible hierarchical compositions of objects, scenes and events as well as their interactions and mutual contexts, and specifies the prior probabilistic distribution of the parse graphs. We present a probabilistic generative model for joint parsing that captures the relations between the input video/text, their corresponding parse graphs and the joint parse graph. Based on the probabilistic model, we propose a joint parsing system consisting of three modules: video parsing, text parsing and joint inference. Video parsing and text parsing produce two parse graphs from the input video and text respectively. The joint inference module produces a joint parse graph by performing matching, deduction and revision on the video and text parse graphs. The proposed framework has the following objectives: Firstly, we aim at deep semantic parsing of video and text that goes beyond the traditional bag-of-words approaches; Secondly, we perform parsing and reasoning across the spatial, temporal and causal dimensions based on the joint S/T/C-AOG representation; Thirdly, we show that deep joint parsing facilitates subsequent applications such as generating narrative text descriptions and answering queries in the forms of who, what, when, where and why. We empirically evaluated our system based on comparison against ground-truth as well as accuracy of query answering and obtained satisfactory results.

Unsupervised Learning of Event AND-OR Grammar and Semantics from Video

Semantic event representation and recognition using syntactic attribute graph grammar

Learning and Parsing Video Events with Goal and Intent Prediction

An Extended Grammar System for Learning and Recognizing Complex Visual Events

Unsupervised Structure Learning of Stochastic And-Or Grammars

Automatic Event Recognition and Anomaly Detection with Attribute Grammar by Learning Scene Semantics

Video Event Representation and Inference on And-Or Graph.

Online semi-supervised learning of composite event rules by combining structure and mass-based predicate similarity

Semi-supervised Early Event Detection.

Joint Video and Text Parsing for Understanding Events and Answering Queries

Controllable Unsupervised Event-Based Video Generation

Learning a Grammar Inducer from Massive Uncurated Instructional Videos

Event-driven Weakly Supervised Video Anomaly Detection

Inferring Social Roles In Long Timespan Video Sequence

Web Video Event Recognition by Semantic Analysis from Ubiquitous Documents

A Generalized Earley Parser for Human Activity Parsing and Prediction.

A Grammatical Compositional Model for Video Action Detection

A Representative-Based Framework For Parsing And Summarizing Events In Surveillance Videos

Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions

Enhancing Video Event Recognition Using Automatically Constructed Semantic-Visual Knowledge Base.

A generic framework for event detection in various video domains.