Abstract:Convolutional neural networks and vision transformers have achieved outstanding performance in machine perception, particularly for image classification. Although these image classifiers excel at predicting image-level class labels, they may not discriminate missing or shifted parts within an object. As a result, they may fail to detect corrupted images that involve missing or disarrayed semantic information in the object composition. On the contrary, human perception easily distinguishes such corruptions. To mitigate this gap, we introduce the concept of "image grammar", consisting of "image semantics" and "image syntax", to denote the semantics of parts or patches of an image and the order in which these parts are arranged to create a meaningful object. To learn the image grammar relative to a class of visual objects/scenes, we propose a weakly supervised two-stage approach. In the first stage, we use a deep clustering framework that relies on iterative clustering and feature refinement to produce part-semantic segmentation. In the second stage, we incorporate a recurrent bi-LSTM module to process a sequence of semantic segmentation patches to capture the image syntax. Our framework is trained to reason over patch semantics and detect faulty syntax. We benchmark the performance of several grammar learning models in detecting patch corruptions. Finally, we verify the capabilities of our framework in Celeb and SUNRGBD datasets and demonstrate that it can achieve a grammar validation accuracy of 70 to 90% in a wide variety of semantic and syntactical corruption scenarios.

Semantic Parsing: Syntactic assurance to target sentence using LSTM Encoder CFG-Decoder

Semantic Parsing for English as a Second Language

Coarse-to-Fine Decoding for Neural Semantic Parsing

Exploring Neural Models for Parsing Natural Language into First-Order Logic

Language to Logical Form with Neural Attention

Natural Language Inference Using Lstm Model With Sentence Fusion

Improving Semantic Parsing with Neural Generator-Reranker Architecture

Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning

Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing.

SLFNet: Generating Semantic Logic Forms from Natural Language Using Semantic Probability Graphs

Towards Image Semantics and Syntax Sequence Learning

Semantic Parsing with Syntax- and Table-Aware SQL Generation

Learning Better Sentence Representation with Syntax Information

Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

Syntax Aware LSTM Model for Chinese Semantic Role Labeling

A semantic processing model for sentence understanding based on cognitive learning

Syntax Aware LSTM model for Semantic Role Labeling.

Transformation-Based Learning For Semantic Parsing

Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling

Syntax-aware Neural Semantic Role Labeling

Error Detection for Text-to-SQL Semantic Parsing