Abstract:In this paper, we propose a practical three-dimensional (3D) real-scene reconstruction framework named Deep3D, which is paired with a deep learning based multi-view stereo (MVS) matching model named the adaptive multi-view aggregation matching (Ada-MVS) model, to obtain a 3D textured mesh model from multi-view oblique aerial images. Deep3D is the first deep learning based framework for 3D scene reconstruction, in which aerial triangulation and view selection are first performed on the input images, and the depth map of each image is then inferred using the pretrained Ada-MVS model. All the inferred depth maps are then fused into a dense point cloud after filtering the outliers. Finally, the 3D textured mesh is extracted from the dense 3D points as the final product. In the Ada-MVS model, a novel adaptive inter-view aggregation module is specially proposed to address the inconsistent information among oblique views and to fuse the multi-view costs into a robust cost volume. A lightweight recurrent regularization module is also designed for high-efficiency processing of high-capacity aerial images with large depth variations. Moreover, as oblique aerial image datasets are currently lacking, we built a large-scale synthetic multi-view oblique aerial image dataset (WHU-OMVS dataset) for deep learning based model training and methodology evaluation for the task of 3D scene reconstruction. The experimental results show that, firstly, the proposed Ada-MVS model has obvious advantages when used with high-capacity oblique aerial images, compared with several relevant learning-based MVS methods. Secondly, through a comprehensive comparison with popular commercial software packages and open-source solutions, it is shown that the proposed Deep3D framework outperforms all the other solutions in terms of reconstruction quality, and outperforms all the open-source solutions and some of the software packages in terms of efficiency on the WHU-OMVS dataset. Thirdly, the Deep3D framework shows a stable generalization ability and excellent performance when applied to other oblique or nadir aerial images, without any further fine-tuning. The dataset and code will be available at http://gpcv.whu.edu.cn/data .

3D Reconstruction of Typical Entities Based on Multi-Perspective Images

Hybrid-MVS: Robust Multi-View Reconstruction with Hybrid Optimization of Visual and Depth Cues

Efficient 3D reconstruction for urban scenes

Robust 3D Reconstruction with an RGB-D Camera

Refined 3D Modeling of Complex Models Based on Stereo Vision

Multi-View Depth Map Sampling for 3D Reconstruction of Natural Scene

Outdoor Scene 3D Reconstruction from Multiple Point Cloud

Visual Odometry Based 3D-Reconstruction

The 3D Reconstruction of Face Model with Active Structured Light and Stereo Vision Fusion

3D Reconstruction From Monocular Images Based on Deep Convolutional Networks

Deep learning based multi-view stereo matching and 3D scene reconstruction from oblique aerial images

Three-Dimensional Reconstruction from Single Image Base on Combination of CNN and Multi-Spectral Photometric Stereo

Multi-View Stereo Via Geometric Expansion And Depth Refinement

Enhanced multi view 3D reconstruction with improved MVSNet

DRI-MVSNet: A Depth Residual Inference Network for Multi-View Stereo Images

3D-object reconstruction and point cloud sculpture based on Kinect

Toward 3D Object Reconstruction from Stereo Images

An Efficient Dense Reconstruction Algorithm from LiDAR and Monocular Camera

LNMVSNet: A Low-Noise Multi-View Stereo Depth Inference Method for 3D Reconstruction

Better Together: Shading Cues and Multi-View Stereo for Reconstruction Depth Optimization

Attention Aware Cost Volume Pyramid Based Multi-view Stereo Network for 3D Reconstruction