Abstract:We present a method for teaching machines to understand and model the underlying spatial common sense of diverse human-object interactions in 3D in a self-supervised way. This is a challenging task, as there exist specific manifolds of the interactions that can be considered human-like and natural, but the human pose and the geometry of objects can vary even for similar interactions. Such diversity makes the annotating task of 3D interactions difficult and hard to scale, which limits the potential to reason about that in a supervised way. One way of learning the 3D spatial relationship between humans and objects during interaction is by showing multiple 2D images captured from different viewpoints when humans interact with the same type of objects. The core idea of our method is to leverage a generative model that produces high-quality 2D images from an arbitrary text prompt input as an "unbounded" data generator with effective controllability and view diversity. Despite its imperfection of the image quality over real images, we demonstrate that the synthesized images are sufficient to learn the 3D human-object spatial relations. We present multiple strategies to leverage the synthesized images, including (1) the first method to leverage a generative image model for 3D human-object spatial relation learning; (2) a framework to reason about the 3D spatial relations from inconsistent 2D cues in a self-supervised manner via 3D occupancy reasoning with pose canonicalization; (3) semantic clustering to disambiguate different types of interactions with the same object types; and (4) a novel metric to assess the quality of 3D spatial learning of interaction.

Learning Object Spatial Relationship from Demonstration

Object-aware Semantic Mapping of Indoor Scenes Using Octomap

Learning Distributional Demonstration Spaces for Task-Specific Cross-Pose Estimation

Spatial Relation Learning in Complementary Scenarios with Deep Neural Networks.

Spatial Relationship Representation for Visual Object Searching

Learning Generalizable 3D Manipulation With 10 Demonstrations

CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images

Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifiers

RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

Learning Occupancy for Monocular 3D Object Detection

Object Goal Visual Navigation Using Semantic Spatial Relationships.

Latent Space Planning for Multiobject Manipulation With Environment-Aware Relational Classifiers

Monocular Human-Object Reconstruction in the Wild

Interpretable Latent Spaces for Learning from Demonstration

LEMON: Learning 3D Human-Object Interaction Relation from 2D Images

Structured Spatial Reasoning with Open Vocabulary Object Detectors

Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detection

Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views

Understanding Spatial Relations through Multiple Modalities

Grounding Object Relations in Language-Conditioned Robotic Manipulation with Semantic-Spatial Reasoning