Abstract:With the increasing reliance of self-driving and similar robotic systems on robust 3D vision, the processing of LiDAR scans with deep convolutional neural networks has become a trend in academia and industry alike. Prior attempts on the challenging Semantic Scene Completion task - which entails the inference of dense 3D structure and associated semantic labels from "sparse" representations - have been, to a degree, successful in small indoor scenes when provided with dense point clouds or dense depth maps often fused with semantic segmentation maps from RGB images. However, the performance of these systems drop drastically when applied to large outdoor scenes characterized by dynamic and exponentially sparser conditions. Likewise, processing of the entire sparse volume becomes infeasible due to memory limitations and workarounds introduce computational inefficiency as practitioners are forced to divide the overall volume into multiple equal segments and infer on each individually, rendering real-time performance impossible. In this work, we formulate a method that subsumes the sparsity of large-scale environments and present S3CNet, a sparse convolution based neural network that predicts the semantically completed scene from a single, unified LiDAR point cloud. We show that our proposed method outperforms all counterparts on the 3D task, achieving state-of-the art results on the SemanticKITTI benchmark. Furthermore, we propose a 2D variant of S3CNet with a multi-view fusion strategy to complement our 3D network, providing robustness to occlusions and extreme sparsity in distant regions. We conduct experiments for the 2D semantic scene completion task and compare the results of our sparse 2D network against several leading LiDAR segmentation models adapted for bird's eye view segmentation on two open-source datasets.

City-scale Continual Neural Semantic Mapping with Three-Layer Sampling and Panoptic Representation

LISNeRF Mapping: LiDAR-based Implicit Mapping via Semantic Neural Fields for Large-Scale 3D Scenes

3D LiDAR Mapping in Dynamic Environments Using a 4D Implicit Neural Representation

Object-aware Semantic Mapping of Indoor Scenes Using Octomap

Large-Scale 3D Semantic Mapping Using Monocular Vision

DeepMIF: Deep Monotonic Implicit Fields for Large-Scale LiDAR 3D Mapping

Semantic Scene Mapping with Spatio-temporal Deep Neural Network for Robotic Applications

Building and optimization of 3D semantic map based on Lidar and camera fusion

Towards a Meaningful 3D Map Using a 3D Lidar and a Camera

Large-scale 3D Semantic Mapping Using Stereo Vision

3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame

Multi-Scale Point-Wise Convolutional Neural Networks for 3D Object Segmentation From LiDAR Point Clouds in Large-Scale Environments

Multimodal sensor-based semantic 3D mapping for a large-scale environment

3d Semantic Mapping Based On Convolutional Neural Networks

Utilization of Semantic Planes: Improved Localization and Dense Semantic Map for Monocular SLAM in Urban Environment

Continual Neural Mapping: Learning an Implicit Scene Representation from Sequential Observations

Swift-Mapping: Online Neural Implicit Dense Mapping in Urban Scenes

Sparse-to-Continuous: Enhancing Monocular Depth Estimation using Occupancy Maps

Semi-supervised Implicit Scene Completion from Sparse LiDAR

ImpliCity: City Modeling from Satellite Images with Deep Implicit Occupancy Fields

S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds