Image-to-Image Translation-Based Structural Damage Data Augmentation for Infrastructure Inspection Using Unmanned Aerial Vehicle

Gi-Hun Gwon,Jin-Hwan Lee,In-Ho Kim,Seung-Chan Baek,Hyung-Jo Jung

DOI: https://doi.org/10.3390/drones7110666

IF: 5.532

2023-11-08

Drones

Abstract:As technology advances, the use of unmanned aerial vehicles (UAVs) and image sensors for structural monitoring and diagnostics is becoming increasingly critical. This approach enables the efficient inspection and assessment of structural conditions. Furthermore, the integration of deep learning techniques has been proven to be highly effective in detecting damage from structural images, as demonstrated in our study. To enable effective learning by deep learning models, a substantial volume of data is crucial, but collecting appropriate instances of structural damage from real-world scenarios poses challenges and demands specialized knowledge, as well as significant time and resources for labeling. In this study, we propose a methodology that utilizes a generative adversarial network (GAN) for image-to-image translation, with the objective of generating synthetic structural damage data to augment the dataset. Initially, a GAN-based image generation model was trained using paired datasets. When provided with a mask image, this model generated an RGB image based on the annotations. The subsequent step generated domain-specific mask images, a critical task that improved the data augmentation process. These mask images were designed based on prior knowledge to suit the specific characteristics and requirements of the structural damage dataset. These generated masks were then used by the GAN model to produce new RGB image data incorporating various types of damage. In the experimental validation conducted across the three datasets to assess the image generation for data augmentation, our results demonstrated that the generated images closely resembled actual images while effectively conveying information about the newly introduced damage. Furthermore, the experimental validation of damage detection with augmented data entailed a comparative analysis between the performance achieved solely with the original dataset and that attained with the incorporation of additional augmented data. The results for damage detection consistently demonstrated that the utilization of augmented data enhanced performance when compared to relying solely on the original images.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the insufficiency of structural damage data obtained by using unmanned aerial vehicles (UAVs) in infrastructure inspection. Specifically, the paper focuses on how to generate synthetic structural damage data through image - to - image translation techniques to augment the existing data sets. This is mainly because the actual collection of structural damage instances in real - world scenarios is often challenging, requires professional knowledge, and is time - consuming and labor - intensive. By generating more synthetic data, the training effect of deep - learning models can be improved, thereby detecting structural damage more effectively.

The paper proposes a method based on generative adversarial networks (GANs) for image - to - image translation, aiming to generate synthetic structural damage data. This method first trains a GAN - based image - generation model through paired data sets. This model can generate RGB images according to the provided mask images. Then, prior knowledge is used to generate domain - specific mask images, which are designed to meet the specific characteristics and requirements of the structural damage data set. The generated mask images are then used by the GAN model to produce new RGB image data, including various types of damage.

Through experimental verification, the images generated by this method are very close to the actual images and can effectively convey the newly introduced damage information. In addition, the performance evaluation of damage detection using the augmented data shows that, after adding additional augmented data, the detection performance has been significantly improved compared to using only the original data. This indicates that the synthetic data generated by the data - augmentation method helps to improve the performance of deep - learning models in structural - damage - detection tasks.