Abstract:This paper presents a new perspective for 3D scene understanding by reasoning object stability and safety using intuitive mechanics. Our approach utilizes a simple observation that, by human design, objects in static scenes should be stable in the gravity field and be safe with respect to various physical disturbances such as human activities. This assumption is applicable to all scene categories and poses useful constraints for the plausible interpretations (parses) in scene understanding. Given a 3D point cloud captured for a static scene by depth cameras, our method consists of three steps: (i) recovering solid 3D volumetric primitives from voxels; (ii) reasoning stability by grouping the unstable primitives to physically stable objects by optimizing the stability and the scene prior; and (iii) reasoning safety by evaluating the physical risks for objects under physical disturbances, such as human activity, wind or earthquakes. We adopt a novel intuitive physics model and represent the energy landscape of each primitive and object in the scene by a disconnectivity graph (DG). We construct a contact graph with nodes being 3D volumetric primitives and edges representing the supporting relations. Then we adopt a Swendson–Wang Cuts algorithm to partition the contact graph into groups, each of which is a stable object. In order to detect unsafe objects in a static scene, our method further infers hidden and situated causes (disturbances) in the scene, and then introduces intuitive physical mechanics to predict possible effects (e.g., falls) as consequences of the disturbances. In experiments, we demonstrate that the algorithm achieves a substantially better performance for (i) object segmentation, (ii) 3D volumetric recovery, and (iii) scene understanding with respect to other state-of-the-art methods. We also compare the safety prediction from the intuitive mechanics model with human judgement.

Table-Top Scene Analysis Using Knowledge-Supervised MCMC

Scene Dynamics: Counterfactual Critic Multi-Agent Training for Scene Graph Generation.

Counterfactual Critic Multi-Agent Training for Scene Graph Generation

V-PRISM: Probabilistic Mapping of Unknown Tabletop Scenes

3D Scene Graph Prediction on Point Clouds Using Knowledge Graphs

Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation

Probabilistic Graph Based Spatial Assembly Relation Inference for Programming of Assembly Task by Demonstration.

Adaptive Image-to-Video Scene Graph Generation via Knowledge Reasoning and Adversarial Learning

Scene Understanding by Reasoning Stability and Safety

Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI

Multi-Modal 3D Scene Graph Updater for Shared and Dynamic Environments

Knowledge-Embedded Routing Network for Scene Graph Generation

TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes

Learning a Probabilistic Topology Discovering Model for Scene Categorization.

Modeling Dynamic Environments with Scene Graph Memory

Reasoning Geometric Commonsense for Single-view 3D Scene Parsing

Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense

Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies

Bayesian Reconstruction of 3d Shapes and Scenes from A Single Image

Scene parsing by data-driven cluster sampling

Beyond Point Clouds: Scene Understanding by Reasoning Geometry and Physics