Abstract:In this paper, we propose a practical three-dimensional (3D) real-scene reconstruction framework named Deep3D, which is paired with a deep learning based multi-view stereo (MVS) matching model named the adaptive multi-view aggregation matching (Ada-MVS) model, to obtain a 3D textured mesh model from multi-view oblique aerial images. Deep3D is the first deep learning based framework for 3D scene reconstruction, in which aerial triangulation and view selection are first performed on the input images, and the depth map of each image is then inferred using the pretrained Ada-MVS model. All the inferred depth maps are then fused into a dense point cloud after filtering the outliers. Finally, the 3D textured mesh is extracted from the dense 3D points as the final product. In the Ada-MVS model, a novel adaptive inter-view aggregation module is specially proposed to address the inconsistent information among oblique views and to fuse the multi-view costs into a robust cost volume. A lightweight recurrent regularization module is also designed for high-efficiency processing of high-capacity aerial images with large depth variations. Moreover, as oblique aerial image datasets are currently lacking, we built a large-scale synthetic multi-view oblique aerial image dataset (WHU-OMVS dataset) for deep learning based model training and methodology evaluation for the task of 3D scene reconstruction. The experimental results show that, firstly, the proposed Ada-MVS model has obvious advantages when used with high-capacity oblique aerial images, compared with several relevant learning-based MVS methods. Secondly, through a comprehensive comparison with popular commercial software packages and open-source solutions, it is shown that the proposed Deep3D framework outperforms all the other solutions in terms of reconstruction quality, and outperforms all the open-source solutions and some of the software packages in terms of efficiency on the WHU-OMVS dataset. Thirdly, the Deep3D framework shows a stable generalization ability and excellent performance when applied to other oblique or nadir aerial images, without any further fine-tuning. The dataset and code will be available at http://gpcv.whu.edu.cn/data .

Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images

Semantic Reconstruction based on RGB Image and Sparse Depth

Semantic 3d Reconstruction Using Multi-View High-Resolution Satellite Images Based On U-Net And Image-Guided Depth Fusion

Semantic Segmentation for High-Resolution Aerial Imagery Using Multi-Skip Network and Markov Random Fields

S3M: Semantic Segmentation Sparse Mapping for UAVs with RGB-D Camera

Visual Consistency Enhancement for Multi-view Stereo Reconstruction in Remote Sensing

A Multi-Step Fusion Network for Semantic Segmentation of High-Resolution Aerial Images

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning

Multiview Fusion Driven 3-D Point Cloud Semantic Segmentation Based on Hierarchical Transformer

SLAM and 3D Semantic Reconstruction Based on the Fusion of Lidar and Monocular Vision

Large-Scale 3D Semantic Mapping Using Monocular Vision

Fusion of images and point clouds for the semantic segmentation of large-scale 3D scenes based on deep learning

A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-View Stereo Reconstruction From an Open Aerial Dataset

Semantic-Driven Informed Planning and 3D Reconstruction for the Quadrotor Unmanned Aerial Vehicle

Real-Time Semantic Plane Reconstruction on a Monocular Drone Using Sparse Fusion

Mmwave radar and vision fusion for semantic 3D reconstruction

Joint 2D and 3D Semantic Segmentation with Consistent Instance Semantic

Semantic Dense Reconstruction with Consistent Scene Segments

Robust 3D Semantic Segmentation Based on Multi-Phase Multi-Modal Fusion for Intelligent Vehicles

Deep learning based multi-view stereo matching and 3D scene reconstruction from oblique aerial images

3D Scene Reconstruction with Sparse LiDAR Data and Monocular Image in Single Frame