Abstract:The reconstruction of missing pixels is essential for remote sensing images, as they often suffer from problems such as covering, dead pixels, and scan line corrector (SLC)-off. Image inpainting techniques can solve these problems, as they can generate realistic content for the unknown regions of an image based on the known regions. Recently, convolutional neural network (CNN)-based inpainting methods have integrated the attention mechanism to improve inpainting performance, as they can capture long-range dependencies and adapt to inputs in a flexible manner. However, to obtain the attention map for each feature, they compute the similarities between the feature and the entire feature map, which may introduce noise from irrelevant features. To address this problem, we propose a novel adaptive attention (Ada-attention) that uses an offset position subnet to adaptively select the most relevant keys and values based on self-attention. This enables the attention to be focused on essential features and model more informative dependencies on the global range. Ada-attention first employs an offset subnet to predict offset position maps on the query feature map; then, it samples the most relevant features from the input feature map based on the offset position; next, it computes key and value maps for self-attention using the sampled features; finally, using the query, key and value maps, the self-attention outputs the reconstructed feature map. Based on Ada-attention, we customized a u-shaped adaptive-attention completing network (AACNet) to reconstruct missing regions. Experimental results on several digital remote sensing and natural image datasets, using two image inpainting models and two remote sensing image reconstruction approaches, demonstrate that the proposed AACNet achieves a good quantitative performance and good visual restoration results with regard to object integrity, texture/edge detail, and structural consistency. Ablation studies indicate that Ada-attention outperforms self-attention in terms of PSNR by 0.66%, SSIM by 0.74%, and MAE by 3.9%, and can focus on valuable global features using the adaptive offset subnet. Additionally, our approach has also been successfully applied to remove real clouds in remote sensing images, generating credible content for cloudy regions.

Preliminary Investigation on Single Remote Sensing Image Inpainting Through a Modified GAN

UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation

Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

Free-Form Image Inpainting with Separable Gate Encoder-Decoder Network

Face Image Inpainting Based on Generative Adversarial Network

The Improved Image Inpainting Algorithm Via Encoder and Similarity Constraint

A Context-Based Multi-Scale Discriminant Model for Natural Image Inpainting

Semantic Image Inpainting with Multi-Stage Feature Reasoning Generative Adversarial Network

Image Inpainting Based on Contextual Coherent Attention GAN

Adaptive-Attention Completing Network for Remote Sensing Image

Panoramic Image Inpainting With Gated Convolution And Contextual Reconstruction Loss

High-Resolution Image Inpainting Based On Multi-Scale Neural Network

A Progressive Image Inpainting Algorithm with a Mask Auto-update Branch

Image Inpainting with Bilateral Convolution

Context-Based Multiscale Unified Network for Missing Data Reconstruction in Remote Sensing Images

Image Inpainting Using Wasserstein Generative Adversarial Network

RSINet: Inpainting Remotely Sensed Images Using Triple GAN Framework

An Adaptive Iterative Inpainting Method with More Information Exploration

Context-Aware Semantic Inpainting

Inpainting of Remote Sensing SST Images With Deep Convolutional Generative Adversarial Network.

Image Fine-grained Inpainting.