Abstract:Autoscanning of an unknown environment is the key to many AR/VR and robotic applications. However, autonomous reconstruction with both high efficiency and quality remains a challenging problem. In this work, we propose a reconstruction-oriented autoscanning approach, called ScanBot, which utilizes hierarchical deep reinforcement learning techniques for global region-of-interest (ROI) planning to improve the scanning efficiency and local next-best-view (NBV) planning to enhance the reconstruction quality. Given the partially reconstructed scene, the global policy designates an ROI with insufficient exploration or reconstruction. The local policy is then applied to refine the reconstruction quality of objects in this region by planning and scanning a series of NBVs. A novel mixed 2D-3D representation is designed for these policies, where a 2D quality map with tailored quality channels encoding the scanning progress is consumed by the global policy, and a coarse-to-fine 3D volumetric representation that embodies both local environment and object completeness is fed to the local policy. These two policies iterate until the whole scene has been completely explored and scanned. To speed up the learning of complex environmental dynamics and enhance the agent's memory for spatial-temporal inference, we further introduce two novel auxiliary learning tasks to guide the training of our global policy. Thorough evaluations and comparisons are carried out to show the feasibility of our proposed approach and its advantages over previous methods. Code and data are available at https://github.com/HezhiCao/Scanbot.

Next-Best View Policy for 3D Reconstruction

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

MAP-NBV: Multi-agent Prediction-guided Next-Best-View Planning for Active 3D Object Reconstruction

ScanBot: Autonomous Reconstruction Via Deep Reinforcement Learning.

Bag of Views: An Appearance-Based Approach to Next-Best-View Planning for 3D Reconstruction

A Shadowcasting-Based Next-Best-View Planner for Autonomous 3D Exploration

Take Your Best Shot: Sampling-Based Next-Best-View Planning for Autonomous Photography & Inspection

Automatic View Planning with Multi-scale Deep Reinforcement Learning Agents

3D Reconstruction in Noisy Agricultural Environments: A Bayesian Optimization Perspective for View Planning

An Efficient Projection-Based Next-best-view Planning Framework for Reconstruction of Unknown Objects

Object-Aware View Planning for Autonomous 3-D Model Reconstruction of Buildings Using a Mobile Robot

Active Implicit Object Reconstruction using Uncertainty-guided Next-Best-View Optimization

Object-aware Guidance for Autonomous Scene Reconstruction

Autonomous Implicit Indoor Scene Reconstruction with Frontier Exploration

A Global Max-Flow-Based Multi-Resolution Next-Best-View Method for Reconstruction of 3D Unknown Objects.

Pred-NBV: Prediction-guided Next-Best-View for 3D Object Reconstruction

Volumetric Information Gain Guided Receding Horizon Planner for Active 3D Reconstruction.

A Double Branch Next-Best-View Network and Novel Robot System for Active Object Reconstruction

Onboard View Planning of a Flying Camera for High Fidelity 3D Reconstruction of a Moving Actor

Fast LiDAR Informed Visual Search in Unseen Indoor Environments

The surface edge explorer (SEE): A measurement-direct approach to next best view planning