Abstract:This paper investigates a fundamental problem of scene understanding: how to parse a scene image into a structured configuration (i.e., a semantic object hierarchy with object interaction relations). We propose a deep architecture consisting of two networks: i) a convolutional neural network (CNN) extracting the image representation for pixel-wise object labeling and ii) a recursive neural network (RsNN) discovering the hierarchical object structure and the inter-object relations. Rather than relying on elaborative annotations (e.g., manually labeled semantic maps and relations), we train our deep model in a weakly-supervised learning manner by leveraging the descriptive sentences of the training images. Specifically, we decompose each sentence into a semantic tree consisting of nouns and verb phrases, and apply these tree structures to discover the configurations of the training images. Once these scene configurations are determined, then the parameters of both the CNN and RsNN are updated accordingly by back propagation. The entire model training is accomplished through an Expectation-Maximization method. Extensive experiments show that our model is capable of producing meaningful scene configurations and achieving more favorable scene labeling results on two benchmarks (i.e., PASCAL VOC 2012 and SYSU-Scenes) compared with other state-of-the-art weakly-supervised deep learning methods. In particular, SYSU-Scenes contains more than 5,000 scene images with their semantic sentence descriptions, which is created by us for advancing research on scene parsing.

Scene Labeling Using Recurrent Neural Networks with Explicit Long Range Contextual Dependency.

Scene Labeling using Gated Recurrent Units with Explicit Long Range Conditioning

DAG-Recurrent Neural Networks for Scene Labeling.

Convolutional Neural Networks with Intra-Layer Recurrent Connections for Scene Labeling

Residual Recurrent Neural Networks for Learning Sequential Representations.

Scene Parsing via Dense Recurrent Neural Networks with Attentional Selection

Fast scene labeling via structural inference

Spatially-Aware Context Neural Networks.

Convolutional recurrent neural networks: Learning spatial dependencies for image representation

Label-Dependencies Aware Recurrent Neural Networks

Multimodal Recurrent Neural Networks with Information Transfer Layers for Indoor Scene Labeling

Scene Segmentation with DAG-Recurrent Neural Networks

Geometric Scene Parsing with Hierarchical LSTM

Context-based conditional random fields as recurrent neural networks for image labeling

LSTM-CF: Unifying Context Modeling and Fusion with LSTMs for RGB-D Scene Labeling

Recurrent Convolutional Neural Networks for Scene Parsing

Chinese Semantic Role Labeling with Bidirectional Recurrent Neural Networks

A holistic representation guided attention network for scene text recognition

Spectral-spatial classification of hyperspectral imagery based on recurrent neural networks

Hierarchical Scene Parsing by Weakly Supervised Learning with Image Descriptions

Towards Accurate Scene Text Recognition with Semantic Reasoning Networks