Abstract:This paper addresses the problem of simultaneous 3D reconstruction and material recognition and segmentation. Enabling robots to recognise different materials (concrete, metal etc.) in a scene is important for many tasks, e.g. robotic interventions in nuclear decommissioning. Previous work on 3D semantic reconstruction has predominantly focused on recognition of everyday domestic objects (tables, chairs etc.), whereas previous work on material recognition has largely been confined to single 2D images without any 3D reconstruction. Meanwhile, most 3D semantic reconstruction methods rely on computationally expensive post-processing, using Fully-Connected Conditional Random Fields (CRFs), to achieve consistent segmentations. In contrast, we propose a deep learning method which performs 3D reconstruction while simultaneously recognising different types of materials and labeling them at the pixel level. Unlike previous methods, we propose a fully end-to-end approach, which does not require hand-crafted features or CRF post-processing. Instead, we use only learned features, and the CRF segmentation constraints are incorporated inside the fully end-to-end learned system. We present the results of experiments, in which we trained our system to perform real-time 3D semantic reconstruction for 23 different materials in a real-world application. The run-time performance of the system can be boosted to around 10Hz, using a conventional GPU, which is enough to achieve realtime semantic reconstruction using a 30fps RGB-D camera. To the best of our knowledge, this work is the first real-time end-to-end system for simultaneous 3D reconstruction and material recognition.

A Hardware-adaptive Deep Feature Matching Pipeline for Real-time 3D Reconstruction.

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

Real-time Underwater 3D Reconstruction Based on Monocular Image

Full Hd Real-Time Depth Estimation Algorithm and Hardware Implementation

Neural Super-Resolution in Real-Time Rendering Using Auxiliary Feature Enhancement

Feature Points Matching for Face Reconstruction Based on the Window Unique Property of Pseudo-Random Coded Image

GPU Accelerated Cascade Hashing Image Matching for Large Scale 3D Reconstruction

A Hardware Architecture of Feature Extraction for Real-Time Visual SLAM.

GPU Accelerated Robust Scene Reconstruction

SDFMAP: Neural Signed Distance Fields for Mapping and Positioning in Real-Time

A Feature Matching Method Based on the Convolutional Neural Network

Depth-Based 3D Face Reconstruction and Pose Estimation Using Shape-Preserving Domain Adaptation

Deep-learning Based Single-Shot 3D Reconstruction with Simulated Color-Crosstalk and Randomized Extrinsics

A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition

Real-time 3d reconstruction of dynamic scenes with multiple kinect v2 sensors

GPU-accelerated Feature Tracking for 3D Reconstruction

Fast and Adaptive 3d Reconstruction with Extensively High Completeness

Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination

Real-time High-accuracy Three-Dimensional Reconstruction with Consumer RGB-D Cameras

Realtime 3D reconstruction algorithm based on improved AKAZE and multi-layer feature detection network

Hardware accelerator for an accurate local stereo matching algorithm using binary neural network