Towards CGAN-based Satellite Image Synthesis with Partial Pixel-Wise Annotation

Hadi Mansourifar,Steven J. Simske

2023-02-02

Abstract:Conditional Generative Adversarial Nets (CGANs) need a significantly huge dataset with a detailed pixel-wise annotation to generate high-quality images. Unfortunately, any amount of missing pixel annotations may significantly impact the result not only locally, but also in annotated areas. To the best of our knowledge, such a challenge has never been investigated in the broader field of GANs. In this paper, we take the first step in this direction to study the problem of CGAN-based satellite image synthesis given partially annotated images. We first define the problem of image synthesis using partially annotated data, and we discuss a scenario in which we face such a challenge. We then propose an effective solution called detail augmentation to address this problem. To do so, we tested two different approaches to augment details to compensate for missing pixel-wise annotations. In the first approach, we augmented the original images with their Canny edges to using the CGAN to compensate for the missing annotations. The second approach, however, attempted to assign a color to all pixels with missing annotation. Eventually, a different CGAN was trained to translate the new feature images into a final output.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper mainly focuses on the issue of Partial Pixel-wise Annotation (PPA) in satellite image synthesis using Conditional Generative Adversarial Networks (CGANs). When generating high-quality images with CGANs, a large dataset with detailed pixel-wise annotations is usually required. However, in practical applications, there are often cases of incomplete pixel-wise annotations, which can affect not only the quality of the generated local areas but also the quality of the annotated areas.

To address this issue, the authors propose two methods to enhance image details to compensate for the missing pixel-wise annotations:

1. **Partial Detail Augmentation (PDA)**: By training a Pix2pix model to convert the input image into its Canny edge feature map, and overlaying the original color pixel-wise annotations on this feature map. Then, another Pix2pix model is trained to transform this new feature map, which has integrated color and edge information, into the final output image.

2. **Full Detail Augmentation (FDA)**: Similar to PDA, but further assigns specific colors to all pixels missing color annotations to achieve the maximum detail level of the background scene. Likewise, the transformation is done through training and converting with two Pix2pix models.

Experimental results show that both methods significantly improve the target detection scores, especially when evaluated with independent target detection models such as AWS Rekognition, Google Vision Cloud, and YOLOv3. Among them, PDA performs better than FDA in most cases, particularly in reducing false positives. In addition, the paper also discusses in detail the performance differences of different models in various scenarios and summarizes the advantages of the two proposed methods.