Abstract:The International Journal of Robotics Research, Ahead of Print. 3D spatial perception is the problem of building and maintaining an actionable and persistent representation of the environment in real-time using sensor data and prior knowledge. Despite the fast-paced progress in robot perception, most existing methods either build purely geometric maps (as in traditional SLAM) or "flat" metric-semantic maps that do not scale to large environments or large dictionaries of semantic labels. The first part of this paper is concerned with representations: we show that scalable representations for spatial perception need to be hierarchical in nature. Hierarchical representations are efficient to store, and lead to layered graphs with small treewidth, which enable provably efficient inference. We then introduce an example of hierarchical representation for indoor environments, namely a 3D scene graph, and discuss its structure and properties. The second part of the paper focuses on algorithms to incrementally construct a 3D scene graph as the robot explores the environment. Our algorithms combine 3D geometry (e.g., to cluster the free space into a graph of places), topology (to cluster the places into rooms), and geometric deep learning (e.g., to classify the type of rooms the robot is moving across). The third part of the paper focuses on algorithms to maintain and correct 3D scene graphs during long-term operation. We propose hierarchical descriptors for loop closure detection and describe how to correct a scene graph in response to loop closures, by solving a 3D scene graph optimization problem. We conclude the paper by combining the proposed perception algorithms into Hydra, a real-time spatial perception system that builds a 3D scene graph from visual-inertial data in real-time. We showcase Hydra's performance in photo-realistic simulations and real data collected by a Clearpath Jackal robots and a Unitree A1 robot. We release an open-source implementation of Hydra at https://github.com/MIT-SPARK/Hydra.

Combining top-down spatial reasoning and bottom-up object class recognition for scene understanding

Multi-Class Detection and Segmentation of Objects in Depth

Foundations of spatial perception for robotics: Hierarchical representations and real-time systems

On Object Perception in Enhanced Reality Environment for Robot Telecontrol

Object-aware Semantic Mapping of Indoor Scenes Using Octomap

An Ontology-Based Framework for Complex Urban Object Recognition Through Integrating Visual Features and Interpretable Semantics.

Using Scale Coordination and Semantic Information for Robust 3-D Object Recognition by a Service Robot

Augmented Environment Representations with Complete Object Models

Towards Scene Understanding with Detailed 3D Object Representations

A Gestaltist approach to contour-based object recognition: Combining bottom-up and top-down cues

Spatial Relation Learning in Complementary Scenarios with Deep Neural Networks.

Latent Space Planning for Multiobject Manipulation With Environment-Aware Relational Classifiers

Identification of Unmodeled Objects from Symbolic Descriptions

Open-Ended Fine-Grained 3D Object Categorization by Combining Shape and Texture Features in Multiple Colorspaces

Structured Spatial Reasoning with Open Vocabulary Object Detectors

Intelligent Spatial Perception by Building Hierarchical 3D Scene Graphs for Indoor Scenarios with the Help of LLMs*

Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognition

Simultaneous Recognition and Modeling for Learning 3-D Object Models From Everyday Scenes.

Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifiers

Persistent Homology Meets Object Unity: Object Recognition in Clutter

Indoor Scene Recognition in 3D