Abstract:Real-time reconstruction of 3D environment attributed with semantic information is significant for a variety of applications, such as obstacle detection, traffic scene comprehension and autonomous navigation. The current approaches to achieve it are mainly using stereo vision, Structure from Motion (SfM) or mobile LiDAR sensors. Each of these approaches has its own limitation, stereo vision has high computational cost, SfM needs accurate calibration between a sequences of images, and the onboard LiDAR sensor can only provide sparse points without color information. This paper describes a novel method for traffic scene semantic segmentation by combining sparse LiDAR point cloud (e.g. from Velodyne scans), with monocular color image. The key novelty of the method is the semantic coupling of stereoscopic point cloud with color lattice from camera image labelled through a Convolutional Neural Network (CNN). The presented method comprises three main process: (I) perform semantic segmentation on color image from monocular camera by using CNN, (II) extract ideal surfaces and other structural information from point cloud, (HI) improve the image segmentation with the extracts and label the point cloud with the image segments. The whole process is done in a single frame, and the output of the system is labelled point cloud which can be used in construction of semantic object convex and alignment between frames. We demonstrate the effectiveness of our system on the KITTI dataset providing sufficient camera and LiDAR data, and present qualitative and quantitative results indicating the improvements in segmentation comparing to methods merely using either image or LiDAR data.

Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask

Single-Mask Inpainting for Voxel-Based Neural Radiance Fields

Learning 3 D Scene Synthesis from Annotated RGB-D Images

Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Novel 3D-Aware Composition Images Synthesis for Object Display with Diffusion Model.

Point-Based Neural Scene Rendering for Street Views

Semantic Reconstruction based on RGB Image and Sparse Depth

View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume

PixelSynth: Generating a 3D-Consistent Experience from a Single Image

Realistic Image Synthesis with Configurable 3D Scene Layouts

Point Cloud Scene Completion With Joint Color and Semantic Estimation From Single RGB-D Image

Generative View Synthesis: From Single-view Semantics to Novel-view Images

Real-time High-resolution View Synthesis of Complex Scenes with Explicit 3D Visibility Reasoning

Learning semantic priors for texture-realistic sketch-to-image synthesis

3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame

View Synthesis with Sculpted Neural Points

CompNVS: Novel View Synthesis with Scene Completion

Pano2Room: Novel View Synthesis from a Single Indoor Panorama

FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models

3D Face Reconstruction and Semantic Annotation from Single Depth Image