Abstract:This paper proposes a parsing algorithm for scene understanding which includes four aspects: computing 3D scene layout, detecting 3D objects (e.g. furniture), detecting 2D faces (windows, doors etc.), and segmenting background. In contrast to previous scene labeling work that applied discriminative classifiers to pixels (or super-pixels), we use a generative Stochastic Scene Grammar (SSG). This grammar represents the compositional structures of visual entities from scene categories, 3D foreground/background, 2D faces, to 1D lines. The grammar includes three types of production rules and two types of contextual relations. Production rules: (i) AND rules represent the decomposition of an entity into sub-parts; (ii) OR rules represent the switching among sub-types of an entity; (iii) SET rules represent an ensemble of visual entities. Contextual relations: (i) Cooperative + relations represent positive links between binding entities, such as hinged faces of a object or aligned boxes; (ii) Competitive - relations represents negative links between competing entities, such as mutually exclusive boxes. We design an efficient MCMC inference algorithm, namely Hierarchical cluster sampling, to search in the large solution space of scene configurations. The algorithm has two stages: (i) Clustering: It forms all possible higher-level structures (clusters) from lower-level entities by production rules and contextual relations. (ii) Sampling: It jumps between alternative structures (clusters) in each layer of the hierarchy to find the most probable configuration (represented by a parse tree). In our experiment, we demonstrate the superiority of our algorithm over existing methods on public dataset. In addition, our approach achieves richer structures in the parse tree.

Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction

A Generalized Earley Parser for Human Activity Parsing and Prediction.

Learning to Decode for Future Success

Activity Grammars for Temporal Action Segmentation

Predicting Human Activities Using Stochastic Grammar.

Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing

AMR Parsing as Sequence-to-Graph Transduction

Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

Learning and Parsing Video Events with Goal and Intent Prediction

Image Parsing Via Stochastic Scene Grammar

A Generative Parser with a Discriminative Recognition Algorithm

A Neural Transition-Based Approach for Semantic Dependency Graph Parsing

Neural-Symbolic Inference for Robust Autoregressive Graph Parsing via Compositional Uncertainty Quantification

Semantic Operator Prediction and Applications

An Extended Grammar System for Learning and Recognizing Complex Visual Events

AMR Parsing Via Graph-Sequence Iterative Inference

Learning Algebraic Recombination for Compositional Generalization

Seq2seq Dependency Parsing.

RNN-Based Sequence-Preserved Attention for Dependency Parsing.

Joint Parsing and Generation for Abstractive Summarization

A spatiotemporal grammar network model with wide attention for short-term traffic flow prediction