Abstract:Real-time reconstruction of 3D environment attributed with semantic information is significant for a variety of applications, such as obstacle detection, traffic scene comprehension and autonomous navigation. The current approaches to achieve it are mainly using stereo vision, Structure from Motion (SfM) or mobile LiDAR sensors. Each of these approaches has its own limitation, stereo vision has high computational cost, SfM needs accurate calibration between a sequences of images, and the onboard LiDAR sensor can only provide sparse points without color information. This paper describes a novel method for traffic scene semantic segmentation by combining sparse LiDAR point cloud (e.g. from Velodyne scans), with monocular color image. The key novelty of the method is the semantic coupling of stereoscopic point cloud with color lattice from camera image labelled through a Convolutional Neural Network (CNN). The presented method comprises three main process: (I) perform semantic segmentation on color image from monocular camera by using CNN, (II) extract ideal surfaces and other structural information from point cloud, (HI) improve the image segmentation with the extracts and label the point cloud with the image segments. The whole process is done in a single frame, and the output of the system is labelled point cloud which can be used in construction of semantic object convex and alignment between frames. We demonstrate the effectiveness of our system on the KITTI dataset providing sufficient camera and LiDAR data, and present qualitative and quantitative results indicating the improvements in segmentation comparing to methods merely using either image or LiDAR data.

RGB-DI Images and Full Convolution Neural Network-Based Outdoor Scene Understanding for Mobile Robots.

3D Map Building for Mobile Robot under Unknown Outdoor Environments

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Outdoor scene understanding of mobile robot via multi-sensor information fusion

An RGB-D Fusion Based Semantic Segmentation Algorithm Based on Neighborhood Metric Relations

Visual Odometry Based 3D-Reconstruction

Semantic Reconstruction based on RGB Image and Sparse Depth

Vision-based Mobile Robot's Environment Outdoor Perception

Scene Understanding in a Large Dynamic Environment Through a Laser-Based Sensing.

Object-aware Semantic Mapping of Indoor Scenes Using Octomap

Online Scene Semantic Understanding Based on Sparsely Correlated Network for AR

Large-Scale 3D Semantic Mapping Using Monocular Vision

Real-time Dense 3D Semantic Mapping Using RGB-D Camera

A Flexible Multi-view Multi-modal Imaging System for Outdoor Scenes

Real-time Scene Segmentation Using a Light Deep Neural Network Architecture for Autonomous Robot Navigation on Construction Sites

3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame

Indoor Scene Reconstruction Using a Rotating Device and Multiple RGB-D Cameras

Active Scene Understanding via Online Semantic Reconstruction

Multi-robot collaborative dense scene reconstruction

Geometrical Features based Visual Relocalization for Indoor Service Robot

Change of Scenery: Unsupervised LiDAR Change Detection for Mobile Robots