3D detection of roof sections from a single satellite image and application to LOD2-building reconstruction

Johann Lussange,Mulin Yu,Yuliya Tarabalka,Florent Lafarge

2023-07-12

Abstract:Reconstructing urban areas in 3D out of satellite raster images has been a long-standing and challenging goal of both academical and industrial research. The rare methods today achieving this objective at a Level Of Details $2$ rely on procedural approaches based on geometry, and need stereo images and/or LIDAR data as input. We here propose a method for urban 3D reconstruction named KIBS(\textit{Keypoints Inference By Segmentation}), which comprises two novel features: i) a full deep learning approach for the 3D detection of the roof sections, and ii) only one single (non-orthogonal) satellite raster image as model input. This is achieved in two steps: i) by a Mask R-CNN model performing a 2D segmentation of the buildings' roof sections, and after blending these latter segmented pixels within the RGB satellite raster image, ii) by another identical Mask R-CNN model inferring the heights-to-ground of the roof sections' corners via panoptic segmentation, unto full 3D reconstruction of the buildings and city. We demonstrate the potential of the KIBS method by reconstructing different urban areas in a few minutes, with a Jaccard index for the 2D segmentation of individual roof sections of $88.55\%$ and $75.21\%$ on our two data sets resp., and a height's mean error of such correctly segmented pixels for the 3D reconstruction of $1.60$ m and $2.06$ m on our two data sets resp., hence within the LOD2 precision range.

Computer Vision and Pattern Recognition,Instrumentation and Methods for Astrophysics,Artificial Intelligence

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper aims to address the problem of detecting roof sections from a single satellite image and applying it to LOD2 (Level Of Detail 2) building reconstruction. Specifically, the authors propose a method called KIBS (Keypoints Inference By Segmentation), which has the following two main features:

1. **Fully deep learning-based 3D roof detection**: Unlike previous methods, KIBS uses deep learning techniques to achieve 3D roof section detection.
2. **Requires only one non-orthogonal satellite image as input**: Unlike traditional methods that require stereo images or LIDAR data, KIBS can complete the 3D reconstruction task with just a single ordinary satellite image.

### Research Background and Objectives

With the rapid development of smart cities and intelligent urbanization, digital city models have become important tools for urban planning, environmental analysis, and infrastructure management. 3D city models generated from satellite, aerial, and LIDAR images can provide detailed environmental representations and aid in decision-making processes. However, the unique challenges of these images (such as perspective changes, lighting conditions, etc.) require continuous technological advancements. This paper proposes a new method to achieve LOD2-level 3D building reconstruction using a single satellite image.

### Main Contributions

1. **Two-step model architecture**: The KIBS method includes two steps:
- Step 1: Use the Mask R-CNN model for 2D segmentation to identify the roof sections of buildings.
- Step 2: Fuse the segmented pixels back into the RGB satellite image and use another Mask R-CNN model to infer the heights of the roof corners, thereby achieving complete 3D reconstruction.

2. **Experimental results**: The method was validated on two datasets, achieving 88.55% and 75.21% 2D segmentation Jaccard indices, and height errors of 1.60 meters and 2.06 meters, respectively, all within the LOD2 accuracy range.

Through these innovations, the KIBS method demonstrates its potential for fast and accurate urban 3D reconstruction.