Abstract:The reconstruction of missing pixels is essential for remote sensing images, as they often suffer from problems such as covering, dead pixels, and scan line corrector (SLC)-off. Image inpainting techniques can solve these problems, as they can generate realistic content for the unknown regions of an image based on the known regions. Recently, convolutional neural network (CNN)-based inpainting methods have integrated the attention mechanism to improve inpainting performance, as they can capture long-range dependencies and adapt to inputs in a flexible manner. However, to obtain the attention map for each feature, they compute the similarities between the feature and the entire feature map, which may introduce noise from irrelevant features. To address this problem, we propose a novel adaptive attention (Ada-attention) that uses an offset position subnet to adaptively select the most relevant keys and values based on self-attention. This enables the attention to be focused on essential features and model more informative dependencies on the global range. Ada-attention first employs an offset subnet to predict offset position maps on the query feature map; then, it samples the most relevant features from the input feature map based on the offset position; next, it computes key and value maps for self-attention using the sampled features; finally, using the query, key and value maps, the self-attention outputs the reconstructed feature map. Based on Ada-attention, we customized a u-shaped adaptive-attention completing network (AACNet) to reconstruct missing regions. Experimental results on several digital remote sensing and natural image datasets, using two image inpainting models and two remote sensing image reconstruction approaches, demonstrate that the proposed AACNet achieves a good quantitative performance and good visual restoration results with regard to object integrity, texture/edge detail, and structural consistency. Ablation studies indicate that Ada-attention outperforms self-attention in terms of PSNR by 0.66%, SSIM by 0.74%, and MAE by 3.9%, and can focus on valuable global features using the adaptive offset subnet. Additionally, our approach has also been successfully applied to remove real clouds in remote sensing images, generating credible content for cloudy regions.

Toward Blind-Adaptive Remote Sensing Image Restoration.

Fast non-iterative blind restoration of hyperspectral images with PSFs

CARNet: Context-Aware Residual Learning for JPEG-LS Compressed Remote Sensing Image Restoration

A Method for Remote Sensing Image Restoration Based on the System Degradation Model

Two-stream deep sparse network for accurate and efficient image restoration

Blind Image Restoration Based on Cycle-Consistent Network

Deep Variational Network Toward Blind Image Restoration

Toward Efficient Deep Blind RAW Image Restoration

Attention-Based Real Image Restoration

Adaptive-Attention Completing Network for Remote Sensing Image

Blind Restoration of a Single Real Turbulence-Degraded Image Based on Self-Supervised Learning

Rethinking Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising

Imaging Simulation and Learning-Based Image Restoration for Remote Sensing Time Delay and Integration Cameras

TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising.

Image Restoration Based onMulti-channel Blind Restoration and Sparse Representation Method

A Multiscale Generalized Shrinkage Threshold Network for Image Blind Deblurring in Remote Sensing

Restoring Real-World Images Affected by Varied Degradations Using a Semi-Supervised Domain Adaptation Network

A Multi-scale Generalized Shrinkage Threshold Network for Image Blind Deblurring in Remote Sensing

Degradation-Aware Self-Attention Based Transformer for Blind Image Super-Resolution