[Hospital stress in a plastic surgery department].

C. Nicolétis

Soins

Abstract:

What problem does this paper attempt to address?

Multimodal sensor-based semantic 3D mapping for a large-scale environment

Jongmin Jeong,Tae Sung Yoon,Jin Bae Park

DOI: https://doi.org/10.1016/j.eswa.2018.03.051

IF: 8.5

2018-09-01

Expert Systems with Applications

Abstract:Semantic 3D mapping is one of the most important fields in robotics, and has been used in many applications, such as robot navigation, surveillance, and virtual reality. In general, semantic 3D mapping is mainly composed of 3D reconstruction and semantic segmentation. As these technologies evolve, there has been great progress in semantic 3D mapping in recent years. Furthermore, the number of robotic applications requiring semantic information in 3D mapping to perform high-level tasks has increased, and many studies on semantic 3D mapping have been published. Existing methods use a camera for both 3D reconstruction and semantic segmentation. However, this is not suitable for large-scale environments and has the disadvantage of high computational complexity. To address this problem, we propose a multimodal sensor-based semantic 3D mapping system using a 3D Lidar combined with a camera. In this study, the odometry is obtained by high-precision global positioning system (GPS) and inertial measurement unit (IMU), and it is estimated by iterative closest point (ICP) when a GPS signal is weak. Then, we use the latest 2D convolutional neural network (CNN) for semantic segmentation. To build a semantic 3D map, we integrate the 3D map with semantic information by using coordinate transformation and Bayes’ update scheme. In order to improve the semantic 3D map, we propose a 3D refinement process to correct wrongly segmented voxels and remove traces of moving vehicles in the 3D map. Through experiments on challenging sequences, we demonstrate that our method outperforms state-of-the-art methods in terms of accuracy and intersection over union (IoU). Thus, our method can be used for various applications that require semantic information in 3D map.

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
LESS-Map: Lightweight and Evolving Semantic Map in Parking Lots for Long-term Self-Localization

MingRui Liu,Tang Xinyang,Qian Yeqiang,Chen Jiming,Li Liang

DOI: https://doi.org/10.1109/icra57147.2024.10610168

2024-01-01

Abstract:Precise and long-term stable localization is essential in parking lots for tasks like autonomous driving or autonomous valet parking, etc. Existing methods rely on a fixed and memory-inefficient map, which lacks robust data association approaches. And it is not suitable for precise localization or long-term map maintenance. In this paper, we propose a novel mapping, localization, and map update system based on ground semantic features, utilizing low-cost cameras. We present a precise and lightweight parameterization method to establish improved data association and achieve accurate localization at centimeter-level. Furthermore, we propose a novel map update approach by implementing high-quality data association for parameterized semantic features, allowing continuous map update and refinement during re-localization, while maintaining centimeter-level accuracy. We validate the performance of the proposed method in real-world experiments and compare it against state-of-the-art algorithms. The proposed method achieves an average accuracy improvement of 5cm during the registration process. The generated maps consume only a compact size of 450 KB/km and remain adaptable to evolving environments through continuous update.
A Submap Joining Algorithm for 3D Reconstruction Using an RGB-D Camera Based on Point and Plane Features

Jun Wang,Jingwei Song,Liang Zhao,Shoudong Huang,Rong Xiong

DOI: https://doi.org/10.1016/j.robot.2019.05.002

IF: 3.7

2019-01-01

Robotics and Autonomous Systems

Abstract:In standard point-based methods, the depth measurements of the point features suffer from noise, which will lead to incorrect global structure of the environment. This paper presents a submap joining based SLAM with an RGB-D camera by introducing planes as well as points as features.This work is consisted of two steps: submap building and submap joining. Several adjacent keyframes, with the corresponding small patches, visual feature points, and planes observed from these keyframes, are used to build a submap. We fuse the submaps into a global map in a sequential fashion, such that, the global structure is recovered gradually through plane feature associations and optimization. We also show that the proposed algorithm can handle plane association problem incrementally in submap level, as the plane covariance can be obtained in each submap. The use of submap significantly reduces the computational cost during the optimization process, while keeping all information about planes. The proposed method is validated using both publicly available RGB-D benchmarks and datasets collected by authors. The algorithm can produce accurate trajectories and high quality 3D models on these challenging datasets, which are difficult for existing RGB-D SLAM or SFM algorithms.
Building and optimization of 3D semantic map based on Lidar and camera fusion

Jing Li,Xin Zhang,Jiehao Li,Yanyu Liu,Junzheng Wang

DOI: https://doi.org/10.1016/j.neucom.2020.06.004

IF: 6

2020-10-01

Neurocomputing

Abstract:<p>When considering the robot application of the complex scenarios, the traditional geometric maps are insufficient because of the lack of interactions with the environment. In this paper, a three-dimensional (3D) semantic map with large-scale and accurate integrating Lidar and camera information is presented to achieve real-time road scenes. Firstly, simultaneous localization and mapping (SLAM) is performed to locate the robot position with the multi-sensor fusion of the Lidar and inertial measurement unit (IMU), and the map of the surrounding scenes is constructed while the robot is moving. Moreover, a convolutional neural networks (CNNs)-based semantic segmentation of images is employed to develop the semantic map of the environment. Following the synchronization of the time and space, the sensor fusion of Lidar and camera are used to generate the semantic labeled frame of point clouds and then create a semantic map in term of the posture. Besides, improving the capacity of classification, a higher-order 3D full connection conditional random fields (CRFs) method is utilized to optimize the semantic map. Finally, extensive experiment results evaluated on the KITTI dataset have illustrated the effectiveness of the proposed method.</p>

computer science, artificial intelligence
Large-Scale 3D Semantic Mapping Using Monocular Vision

Cheng Zhang,Zhi Liu,Guangwen Liu,Dandan Huang

DOI: https://doi.org/10.1109/icivc47709.2019.8981035

2019-01-01

Abstract:In recent years, with the rapid development of artificial intelligence, simultaneous localization and mapping technology to solve robot perception problems has been widely used in automatic driving, robot navigation, augmented reality and other fields. In this paper, a method of reconstructing large outdoor 3D dense semantic map based on monocular vision is proposed. Firstly, the motion trajectory of the camera is estimated by visual odometer, and the depth data is obtained by monocular depth estimation algorithm. Meanwhile, a dense conditional random field (CRF) image segmentation system based on deep learning and super-pixel distribution is used for semantic segmentation. Then 2D image semantics are gradually transferred to 3D point clouds by Bayesian progressive label migration strategy and further optimize 3D labels through a novel CRF model. Super pixels are utilized to enforce smoothness and form robust PN high order potential. Finally, dense 3D semantic maps of urban environment is generated by arbitrary length image sequence. The algorithm is tested on KITTI dataset, and the constructed semantic map shows that the proposed algorithm can reconstruct globally consistent semantic map in large-scale outdoor scenes.
Object-aware Semantic Mapping of Indoor Scenes Using Octomap

Kaijian Liu,Zhen Fan,Meiqin Liu,Senlin Zhang

DOI: https://doi.org/10.23919/chicc.2019.8865848

2019-01-01

Abstract:In order to enable robots to provide high-level services in complex indoor environments, it is necessary to improve the robots' ability to cognize the environments. Most of the existing research is focused on indoor 3D reconstruction and semantic segmentation without the organization and maintenance of object recognition results. In this paper, we present an approach to build a 3D semantic map that includes both voxel-based geometrical demonstrations and object-aware entities with the combination of Simultaneous Localization and Mapping (SLAM) and Mask Region-based Convolutional Network (Mask R-CNN). An extended seeded region growing algorithm is designed for 3D segmentation refinement, and an octree-based framework octomap is used to present 3D map in replacement of point cloud map. We present experiments in a simulated home environment and the experimental results verify the accuracy and efficiency of our method.
Semi-Dense 3D Semantic Mapping from Monocular SLAM

Xuanpeng Li,Rachid Belaroussi

DOI: https://doi.org/10.48550/arXiv.1611.04144

2016-11-13

Abstract:The bundle of geometry and appearance in computer vision has proven to be a promising solution for robots across a wide variety of applications. Stereo cameras and RGB-D sensors are widely used to realise fast 3D reconstruction and trajectory tracking in a dense way. However, they lack flexibility of seamless switch between different scaled environments, i.e., indoor and outdoor scenes. In addition, semantic information are still hard to acquire in a 3D mapping. We address this challenge by combining the state-of-art deep learning method and semi-dense Simultaneous Localisation and Mapping (SLAM) based on video stream from a monocular camera. In our approach, 2D semantic information are transferred to 3D mapping via correspondence between connective Keyframes with spatial consistency. There is no need to obtain a semantic segmentation for each frame in a sequence, so that it could achieve a reasonable computation time. We evaluate our method on indoor/outdoor datasets and lead to an improvement in the 2D semantic labelling over baseline single frame predictions.

Computer Vision and Pattern Recognition
DeLS-3D: Deep Localization and Segmentation with a 3D Semantic Map

Peng Wang,Ruigang Yang,Binbin Cao,Wei Xu,Yuanqing Lin

DOI: https://doi.org/10.1109/cvpr.2018.00614

2018-06-01

Abstract:For applications such as augmented reality, autonomous driving, self-localization/camera pose estimation and scene parsing are crucial technologies. In this paper, we propose a unified framework to tackle these two problems simultaneously. The uniqueness of our design is a sensor fusion scheme which integrates camera videos, motion sensors (GPS/IMU), and a 3D semantic map in order to achieve robustness and efficiency of the system. Specifically, we first have an initial coarse camera pose obtained from consumer-grade GPS/IMU, based on which a label map can be rendered from the 3D semantic map. Then, the rendered label map and the RGB image are jointly fed into a pose CNN, yielding a corrected camera pose. In addition, to incorporate temporal information, a multi-layer recurrent neural network (RNN)is further deployed improve the pose accuracy. Finally, based on the pose from RNN, we render a new label map, which is fed together with the RGB image into a segment CNN which produces per-pixel semantic label. In order to validate our approach, we build a dataset with registered 3D point clouds and video camera images. Both the point clouds and the images are semantically-labeled. Each video frame has ground truth pose from highly accurate motion sensors. We show that practically, pose estimation solely relying on images like PoseNet [25] may fail due to street view confusion, and it is important to fuse multiple sensors. Finally, various ablation studies are performed, which demonstrate the effectiveness of the proposed system. In particular, we show that scene parsing and pose estimation are mutually beneficial to achieve a more robust and accurate system.
An Approach for Construct Semantic Map with Scene Classification and Object Semantic Segmentation

Peng Wang,Jun Cheng,Wei Feng

DOI: https://doi.org/10.1109/rcar.2018.8621769

2018-01-01

Abstract:The autonomous navigation system of human beings is more efficient than artificial intelligence technologies, as prior knowledge such as semantic information of the objects and environment is effectively utilized by humans in complicated tasks. Therefore, the ability to build a complete and efficient semantic map system is critical for robot autonomous navigation. In this article, we propose a new framework for building an environment semantic map. Specifically, we construct a 2D semantic map by projecting 3D scene semantic information recognized by convolutional neural network onto a 2D plane. 3D reconstruction of the environment is achieved by RGB-D SLAM 3D space mapping algorithm. We simplify the 3D clustering connectability, using only three cues, to achieve real-time boundary-aware detection of foreground objects. The network trained on the public benchmark dataset is employed to classify the material of the foreground extraction object. Experimental results have demonstrated that the proposed framework is effective in the contraction of semantic map and can be efficiently utilized in the real-time semantic slam system.
Hybrid Semi-Dense 3D Semantic-Topological Mapping From Stereo Visual-Inertial Odometry SLAM With Loop Closure Detection

Shuhuan Wen,Yanfang Zhao,Xin Liu,Fuchun Sun,Huibin Lu,Zhe Wang

DOI: https://doi.org/10.1109/tvt.2020.3041852

IF: 6.8

2020-12-01

IEEE Transactions on Vehicular Technology

Abstract:Intelligent mobile vehicles asks for the ability to learn to comprehend immediate surroundings similar to human cognition and autonomously navigate in an unknown scene. Although simultaneous localization and mapping (SLAM) can construct a geometrically surrounding map, the map contains little semantic information. This paper presents a novel stereo visual-inertial system with loop closure detection based on a semantic-topological map framework. In our system, a hybrid 3D point cloud semantic-topological mapping framework is used to realize autonomous navigation simply by providing a map for path planning and meanwhile for storing semantic information under a dynamic environment. A stereo visual-inertial system can obtain more accurate visual-inertial odometry than a monocular system, and a visual-inertial system is capable of real-time map correction when the localization module detects a loop closure. This paper adopts state-of-the-art Mask R-CNN components to obtain 2D semantic information and project it to a semi-dense 3D semantic-topological map built by stereo visual-inertial SLAM. We perform experimental evaluation of our proposed system on the EuRoC MAV datasets and we also compare it with other competitive methods. Our results demonstrate that the proposed method is more effective than the other competitive methods.

telecommunications,engineering, electrical & electronic,transportation science & technology
Reconstruction of High-Precision Semantic Map

Xinyuan Tu,Jian Zhang,Runhao Luo,Kai Wang,Qingji Zeng,Yu Zhou,Yao Yu,Sidan Du

DOI: https://doi.org/10.3390/s20216264

IF: 3.9

2020-11-03

Sensors

Abstract:We present a real-time Truncated Signed Distance Field (TSDF)-based three-dimensional (3D) semantic reconstruction for LiDAR point cloud, which achieves incremental surface reconstruction and highly accurate semantic segmentation. The high-precise 3D semantic reconstruction in real time on LiDAR data is important but challenging. Lighting Detection and Ranging (LiDAR) data with high accuracy is massive for 3D reconstruction. We so propose a line-of-sight algorithm to update implicit surface incrementally. Meanwhile, in order to use more semantic information effectively, an online attention-based spatial and temporal feature fusion method is proposed, which is well integrated into the reconstruction system. We implement parallel computation in the reconstruction and semantic fusion process, which achieves real-time performance. We demonstrate our approach on the CARLA dataset, Apollo dataset, and our dataset. When compared with the state-of-art mapping methods, our method has a great advantage in terms of both quality and speed, which meets the needs of robotic mapping and navigation.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
From Satellite to Ground: Satellite Assisted Visual Localization with Cross-view Semantic Matching

Guo Xiyue,Peng Haocheng,Hu Junjie,Bao Hujun,Zhang Guofeng

DOI: https://doi.org/10.1109/icra57147.2024.10611658

2024-01-01

Abstract:One of the key challenges of visual Simultaneous Localization and Mapping (SLAM) in large-scale environments is how to effectively use global localization to correct the cumulative errors from long-term tracking. This challenge presents itself in two main aspects: first, the difficulty for robots in revisiting previous locations to perform loop closure, and second, the considerable memory resources required to maintain point-cloud-based global maps. Recent solutions have resorted into neural networks, using satellite images as the references for ground-level localization. However, most of these methods merely provide cross-view patch-matching results, which leads to unfeasible in integration with the SLAM system. To address these issues, we present a semantic-based cross-view localization method. This approach combines semantic information with a reward and penalty mechanism, enabling us to obtain a global probability map and achieve precise 3-degree-of-freedom (3-DoF) localization. Based on that, we develop a SLAM system that capitalizes on satellite imagery for global localization. This strategy effectively bridges the gap between SLAM and real-world coordinates while also substantially reducing accumulated errors. Our experimental results demonstrate that our global localization method significantly outperforms existing satellite-based systems. Moreover, in scenarios where the robot struggles to find loop closures, employing our localization method improves the SLAM accuracy.
Real-time Dense 3D Semantic Mapping Using RGB-D Camera

D. Zhu,Guanci Yang,Xin Eric Wang,Yang Li

DOI: https://doi.org/10.23919/CCC58697.2023.10241093

2023-07-24

Abstract:Focusing on the poor real-time performance of 3D semantic mapping using RGB-D cameras in indoor environments, this paper investigates the Real-time Dense 3D Semantic Mapping (RD3DSM) using RGB-D camera. Firstly, a lightweight semantic point cloud acquisition algorithm based on keyframe is designed, which is used to segment scene images and obtain the semantic point cloud real-timely. Secondly, to address the loss of inter-frame tracking in the semantic mapping process, ORB-SLAM3 system is employed as the underlying architecture of the visual 3D semantic mapping system to provide stable camera pose. Thirdly, an information fusion mechanism of the visual 3D semantic mapping system is designed, and the OctoMap 3D scene reconstruction module is adopted to implement dense 3D map representation with semantic information, and then a visual 3D semantic mapping algorithm based on ORB-SLAM3 is proposed. Finally, the designed RD3DSM system is integrated into a robot platform. The tests were conducted on the TUM dataset and natural indoor scenes. The results show the effectiveness and real-time performance of the RD3DSM system.

Engineering,Computer Science
LISNeRF Mapping: LiDAR-based Implicit Mapping via Semantic Neural Fields for Large-Scale 3D Scenes

Jianyuan Zhang,Zhiliu Yang,Meng Zhang

2024-03-20

Abstract:Large-scale semantic mapping is crucial for outdoor autonomous agents to fulfill high-level tasks such as planning and navigation. This paper proposes a novel method for large-scale 3D semantic reconstruction through implicit representations from posed LiDAR measurements alone. We first leverage an octree-based and hierarchical structure to store implicit features, then these implicit features are decoded to semantic information and signed distance value through shallow Multilayer Perceptrons (MLPs). We adopt off-the-shelf algorithms to predict the semantic labels and instance IDs of point clouds. We then jointly optimize the feature embeddings and MLPs parameters with a self-supervision paradigm for point cloud geometry and a pseudo-supervision paradigm for semantic and panoptic labels. Subsequently, categories and geometric structures for novel points are regressed, and marching cubes are exploited to subdivide and visualize the scenes in the inferring stage. For scenarios with memory constraints, a map stitching strategy is also developed to merge sub-maps into a complete map. Experiments on two real-world datasets, SemanticKITTI and SemanticPOSS, demonstrate the superior segmentation efficiency and mapping effectiveness of our framework compared to current state-of-the-art 3D LiDAR mapping methods.

Computer Vision and Pattern Recognition
Utilization of Semantic Planes: Improved Localization and Dense Semantic Map for Monocular SLAM in Urban Environment

Yaoqi Bao,Yun Pan,Zhe Yang,Ruohong Huan

DOI: https://doi.org/10.1109/lra.2021.3091396

IF: 5.2

2021-07-01

IEEE Robotics and Automation Letters

Abstract:In this letter, we propose a novel semantic direct monocular simultaneous localization and mapping (SLAM) system that fuses the semantic information obtained by an advanced deep neural network (DNN) into direct sparse odometry with loop closure(LDSO), with the purpose of improving the localization accuracy and building a dense semantic map of the urban environment. For localization, we apply a point reselection strategy based on coarse semantic plane (CSP) constraints to discard static points inconsistent with the nearby co-plane points of the same semantic class and dynamic points beyond the visible range. Moreover, a point group movement consistency (PGMC) check is utilized to decrease the impact of moving dynamic objects. For the dense semantic map, we model numerous small semantic planes from well-estimated points to measure the depth of each static pixel, rather than conduct stereo matching. Experimental results show that our method is more accurate than LDSO and comparable with ORB-SLAM in terms of localization. Moreover, it is capable of building a dense semantic map of the urban environment for better scene understanding.

robotics
RS-SLAM: Real time semantic slam with driverless car using LiDAR-Camera-IMU sensing

Chuanwei Zhang,Ruiqi Zhao,Peilin Qin,Jiajia Yang

DOI: https://doi.org/10.1088/1402-4896/ad7a30

2024-09-14

Physica Scripta

Abstract:Accurate and robust Simultaneous Localization and Mapping (SLAM) technology is a critical component of driverless cars, and semantic information plays a vital role in their analysis and understanding of the scene. In the actual scene, the moving object will produce the shadow phenomenon in the mapping process. The positioning accuracy and mapping effect will be affected. Therefore, we propose a semantic SLAM framework combining LiDAR, IMU, and camera, which includes a semantic fusion front-end odometry module and a closed-loop back-end optimization module based on semantic information. An improved image semantic segmentation algorithm based on Deeplabv3+ is designed to enhance the performance of the image semantic segmentation model by replacing the backbone network and introducing an attention mechanism to ensure the accuracy of point cloud segmentation. Dynamic objects are detected and eliminated by calculating the similarity score of semantic labels. A loop closure detection method based on semantic information is proposed to detect key semantic features and use threshold range detection and point cloud re-matching to establish the correct loop closure detection, and finally reduce the global cumulative error and improve the global trajectory accuracy using graph optimization to ultimately obtain the global motion trajectory and realize the construction of 3D semantic maps. We evaluated it on the KITTI dataset and collected a dataset for evaluation by ourselves, which includes four different sequences. The results show that the proposed framework has good positioning accuracy and mapping effect in large-scale urban road environments.

physics, multidisciplinary
Semantic segmentation of surface from lidar point cloud

Aritra Mukherjee,Sourya Dipta Das,Jasorsi Ghosh,Ananda S. Chowdhury,Sanjoy Kumar Saha

DOI: https://doi.org/10.1007/s11042-020-09841-2

IF: 2.577

2020-10-03

Multimedia Tools and Applications

Abstract:Mapping the environment for robot navigation is an important and challenging task in SLAM (Simultaneous Localization And Mapping). Lidar sensor can produce near accurate 3D map of the environment in real time in form of point clouds. Though the point cloud data is adequate for building the map of the environment, processing millions of points in a point cloud is found to be computationally expensive. In this paper, we propose a fast algorithm that can be used to extract semantically labelled surface segments from the cloud in real time for direct navigational use or for higher level contextual scene reconstruction. First, a single scan from a spinning Lidar is used to generate a mesh of sampled cloud points. The generated mesh is further used for surface normal computation of a set of points on the basis of which surface segments are estimated. A novel descriptor is proposed to represent the surface segments. This descriptor is used to determine the surface class (semantic label) of the segments with the help of a classifier. These semantic surface segments can be further utilized for geometric reconstruction of objects in the scene or for optimized trajectory planning of a robot. The proposed method is compared with a number of point cloud segmentation methods and state of the art semantic segmentation methods to demonstrate its efficacy in terms of speed and accuracy.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Cooperative indoor 3D mapping and modeling using LiDAR data

Chenglu Wen,Jinbin Tan,Fashuai Li,Chongrong Wu,Yitai Lin,Zhiyong Wang,Cheng Wang

DOI: https://doi.org/10.1016/j.ins.2021.06.006

IF: 8.1

2021-10-01

Information Sciences

Abstract:<p>Point clouds and models with semantic information facilitate various indoor automation, ranging from indoor robotics to emergency responses. Studies are currently being conducted on semantic labeling and modeling based on offline mapped point clouds, in which, the performance is strongly limited by the mapping process. To address this issue, we propose a framework to cooperatively perform the three tasks of semantic labeling, mapping, and 3D modeling of point clouds. First, our framework uses a deep-learning-assisted method to perform frame-level point cloud semantic labeling. Subsequently, point cloud frames with semantic labels are used to extract the structural planes of buildings, followed by the generation of line structures from the planes. Then, these frames are used to estimate the initial poses of a 3D sensor for data collection. In the subsequent pose optimization process, the initial poses are optimized under the constraints of the structural planes. Finally, the optimized poses are used to integrate semantic frames and line structures to generate a point cloud map and 3D line model of buildings. The experimental results show that the proposed method achieves better results than the state-of-the-art methods that separately perform one of the two tasks.</p>

computer science, information systems
SemanticSLAM: Learning based Semantic Map Construction and Robust Camera Localization

Mingyang Li,Yue Ma,Qinru Qiu

2024-01-24

Abstract:Current techniques in Visual Simultaneous Localization and Mapping (VSLAM) estimate camera displacement by comparing image features of consecutive scenes. These algorithms depend on scene continuity, hence requires frequent camera inputs. However, processing images frequently can lead to significant memory usage and computation overhead. In this study, we introduce SemanticSLAM, an end-to-end visual-inertial odometry system that utilizes semantic features extracted from an RGB-D sensor. This approach enables the creation of a semantic map of the environment and ensures reliable camera localization. SemanticSLAM is scene-agnostic, which means it doesn't require retraining for different environments. It operates effectively in indoor settings, even with infrequent camera input, without prior knowledge. The strength of SemanticSLAM lies in its ability to gradually refine the semantic map and improve pose estimation. This is achieved by a convolutional long-short-term-memory (ConvLSTM) network, trained to correct errors during map construction. Compared to existing VSLAM algorithms, SemanticSLAM improves pose estimation by 17%. The resulting semantic map provides interpretable information about the environment and can be easily applied to various downstream tasks, such as path planning, obstacle avoidance, and robot navigation. The code will be publicly available at

Robotics,Computer Vision and Pattern Recognition
3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame

Yuanxin Zhong,Sijia Wang,Shichao Xie,Zhong Cao,Kun Jiang,Diange Yang

DOI: https://doi.org/10.4271/07-11-01-0005

2017-01-01

Abstract:Real-time reconstruction of 3D environment attributed with semantic information is significant for a variety of applications, such as obstacle detection, traffic scene comprehension and autonomous navigation. The current approaches to achieve it are mainly using stereo vision, Structure from Motion (SfM) or mobile LiDAR sensors. Each of these approaches has its own limitation, stereo vision has high computational cost, SfM needs accurate calibration between a sequences of images, and the onboard LiDAR sensor can only provide sparse points without color information. This paper describes a novel method for traffic scene semantic segmentation by combining sparse LiDAR point cloud (e.g. from Velodyne scans), with monocular color image. The key novelty of the method is the semantic coupling of stereoscopic point cloud with color lattice from camera image labelled through a Convolutional Neural Network (CNN). The presented method comprises three main process: (I) perform semantic segmentation on color image from monocular camera by using CNN, (II) extract ideal surfaces and other structural information from point cloud, (HI) improve the image segmentation with the extracts and label the point cloud with the image segments. The whole process is done in a single frame, and the output of the system is labelled point cloud which can be used in construction of semantic object convex and alignment between frames. We demonstrate the effectiveness of our system on the KITTI dataset providing sufficient camera and LiDAR data, and present qualitative and quantitative results indicating the improvements in segmentation comparing to methods merely using either image or LiDAR data.

[Hospital stress in a plastic surgery department].

Multimodal sensor-based semantic 3D mapping for a large-scale environment

LESS-Map: Lightweight and Evolving Semantic Map in Parking Lots for Long-term Self-Localization

A Submap Joining Algorithm for 3D Reconstruction Using an RGB-D Camera Based on Point and Plane Features

Building and optimization of 3D semantic map based on Lidar and camera fusion

Large-Scale 3D Semantic Mapping Using Monocular Vision

Object-aware Semantic Mapping of Indoor Scenes Using Octomap

Semi-Dense 3D Semantic Mapping from Monocular SLAM

DeLS-3D: Deep Localization and Segmentation with a 3D Semantic Map

An Approach for Construct Semantic Map with Scene Classification and Object Semantic Segmentation

Hybrid Semi-Dense 3D Semantic-Topological Mapping From Stereo Visual-Inertial Odometry SLAM With Loop Closure Detection

Reconstruction of High-Precision Semantic Map

From Satellite to Ground: Satellite Assisted Visual Localization with Cross-view Semantic Matching

Real-time Dense 3D Semantic Mapping Using RGB-D Camera

LISNeRF Mapping: LiDAR-based Implicit Mapping via Semantic Neural Fields for Large-Scale 3D Scenes

Utilization of Semantic Planes: Improved Localization and Dense Semantic Map for Monocular SLAM in Urban Environment

RS-SLAM: Real time semantic slam with driverless car using LiDAR-Camera-IMU sensing

Semantic segmentation of surface from lidar point cloud

Cooperative indoor 3D mapping and modeling using LiDAR data

SemanticSLAM: Learning based Semantic Map Construction and Robust Camera Localization

3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame