Abstract:Real-time reconstruction of 3D environment attributed with semantic information is significant for a variety of applications, such as obstacle detection, traffic scene comprehension and autonomous navigation. The current approaches to achieve it are mainly using stereo vision, Structure from Motion (SfM) or mobile LiDAR sensors. Each of these approaches has its own limitation, stereo vision has high computational cost, SfM needs accurate calibration between a sequences of images, and the onboard LiDAR sensor can only provide sparse points without color information. This paper describes a novel method for traffic scene semantic segmentation by combining sparse LiDAR point cloud (e.g. from Velodyne scans), with monocular color image. The key novelty of the method is the semantic coupling of stereoscopic point cloud with color lattice from camera image labelled through a Convolutional Neural Network (CNN). The presented method comprises three main process: (I) perform semantic segmentation on color image from monocular camera by using CNN, (II) extract ideal surfaces and other structural information from point cloud, (HI) improve the image segmentation with the extracts and label the point cloud with the image segments. The whole process is done in a single frame, and the output of the system is labelled point cloud which can be used in construction of semantic object convex and alignment between frames. We demonstrate the effectiveness of our system on the KITTI dataset providing sufficient camera and LiDAR data, and present qualitative and quantitative results indicating the improvements in segmentation comparing to methods merely using either image or LiDAR data.

SLAM and 3D Semantic Reconstruction Based on the Fusion of Lidar and Monocular Vision

RS-SLAM: Real time semantic slam with driverless car using LiDAR-Camera-IMU sensing

Research on SLAM Algorithm of Mobile Robot Based on the Fusion of 2D LiDAR and Depth Camera

Building and optimization of 3D semantic map based on Lidar and camera fusion

HVL-SLAM: Hybrid Vision and LiDAR Fusion for SLAM

Semantic Mapping Optimization Based on LIDAR and Camera Data Fusion for Autonomous Vehicle

Stereo and LiDAR Loosely Coupled SLAM Constrained Ground Detection

Improving SLAM Techniques with Integrated Multi-Sensor Fusion for 3D Reconstruction

A Monocular Visual SLAM System Augmented by Lightweight Deep Local Feature Extractor Using In-House and Low-Cost LIDAR-camera Integrated Device

DV-LOAM: Direct Visual LiDAR Odometry and Mapping

Dense 3D Semantic SLAM of traffic environment based on stereo vision

LiDAR Odometry and Mapping Based on Semantic Information for Outdoor Environment

Hybrid Semi-Dense 3D Semantic-Topological Mapping From Stereo Visual-Inertial Odometry SLAM With Loop Closure Detection

An Efficient LiDAR SLAM With Angle-Based Feature Extraction and Voxel-Based Fixed-Lag Smoothing

A Novel Lidar-Assisted Monocular Visual SLAM Framework for Mobile Robots in Outdoor Environments

Multi-Sensor Fusion for Robust Simultaneous Localization and Mapping

3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame

SuMa++: Efficient LiDAR-based Semantic SLAM.

SELVO: A Semantic-Enhanced Lidar-Visual Odometry.

Research on SLAM of Unmanned Platform Based on the Fusion of Lidar and Depth Camera