Abstract:The International Journal of Robotics Research, Ahead of Print. 3D spatial perception is the problem of building and maintaining an actionable and persistent representation of the environment in real-time using sensor data and prior knowledge. Despite the fast-paced progress in robot perception, most existing methods either build purely geometric maps (as in traditional SLAM) or "flat" metric-semantic maps that do not scale to large environments or large dictionaries of semantic labels. The first part of this paper is concerned with representations: we show that scalable representations for spatial perception need to be hierarchical in nature. Hierarchical representations are efficient to store, and lead to layered graphs with small treewidth, which enable provably efficient inference. We then introduce an example of hierarchical representation for indoor environments, namely a 3D scene graph, and discuss its structure and properties. The second part of the paper focuses on algorithms to incrementally construct a 3D scene graph as the robot explores the environment. Our algorithms combine 3D geometry (e.g., to cluster the free space into a graph of places), topology (to cluster the places into rooms), and geometric deep learning (e.g., to classify the type of rooms the robot is moving across). The third part of the paper focuses on algorithms to maintain and correct 3D scene graphs during long-term operation. We propose hierarchical descriptors for loop closure detection and describe how to correct a scene graph in response to loop closures, by solving a 3D scene graph optimization problem. We conclude the paper by combining the proposed perception algorithms into Hydra, a real-time spatial perception system that builds a 3D scene graph from visual-inertial data in real-time. We showcase Hydra's performance in photo-realistic simulations and real data collected by a Clearpath Jackal robots and a Unitree A1 robot. We release an open-source implementation of Hydra at https://github.com/MIT-SPARK/Hydra.

A novel visual organization based on topological perception

Visual Complexity of Shapes: a Hierarchical Perceptual Learning Model

An Unified CGA-Based Formal Expression of Spatio-Temporal Topological Relations for Computation and Analysis of Geographic Objects

Saliency Detection Inspired by Topological Perception Theory

Towards Robust Perception Using Topological Invariants

Topological Shape Changes Weaken the Innate Defensive Response to Visual Threat in Mice

Computational Topology and Its Applications in Geometric Design

DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization

Learning Point Cloud Shapes with Geometric and Topological Structures.

Visual relationship detection with region topology structure

High-Order Topology Modeling of Visual Words for Image Classification

Learning a Probabilistic Topology Discovering Model for Scene Categorization.

A topological solution to object segmentation and tracking

Efficient Topological Localization Using Orientation Adjacency Coherence Histograms

Topograph: An efficient Graph-Based Framework for Strictly Topology Preserving Image Segmentation

Exploring Figure-Ground Assignment Mechanism in Perceptual Organization

Visual Orientation Selectivity Based Structure Description

Locating topological structures in digital images via local homology

Attention Selection Using Global Topological Properties Based on Pulse Coupled Neural Network

Principles Governing the Topological Organization of Object Selectivities in Ventral Temporal Cortex

Foundations of spatial perception for robotics: Hierarchical representations and real-time systems