Abstract:Pansharpening is a challenging image fusion task that involves restoring images using two different modalities: low-resolution multispectral images (LRMS) and high-resolution panchromatic (PAN). Many end-to-end specialized models based on deep learning (DL) have been proposed, yet the scale and performance of these models are limited by the size of dataset. Given the superior parameter scales and feature representations of pre-trained models, they exhibit outstanding performance when transferred to downstream tasks with small datasets. Therefore, we propose an efficient fine-tuning method, namely PanAdapter, which utilizes additional advanced semantic information from pre-trained models to alleviate the issue of small-scale datasets in pansharpening tasks. Specifically, targeting the large domain discrepancy between image restoration and pansharpening tasks, the PanAdapter adopts a two-stage training strategy for progressively adapting to the downstream task. In the first stage, we fine-tune the pre-trained CNN model and extract task-specific priors at two scales by proposed Local Prior Extraction (LPE) module. In the second stage, we feed the extracted two-scale priors into two branches of cascaded adapters respectively. At each adapter, we design two parameter-efficient modules for allowing the two branches to interact and be injected into the frozen pre-trained VisionTransformer (ViT) blocks. We demonstrate that by only training the proposed LPE modules and adapters with a small number of parameters, our approach can benefit from pre-trained image restoration models and achieve state-of-the-art performance in several benchmark pansharpening datasets. The code will be available soon.

Structure Tensor-Driven Block-Based Adaptive Variational Pansharpening

An anisotropic variational pansharpening model with adaptive coefficients

SSETPAN: Spatial-Spectral Enhanced Transformer Based Network for Pansharpening

Variational Pansharpening Based on Coefficient Estimation With Nonlocal Regression

Pansharpening by Combining Enhanced Spatial-Spectral Fidelity and Gradient-Domain Guided Filtering

Adaptive Spatial Structure-Aware and Spectral Gradient Structure Tensor-Guided Model for Pansharpening

Convolutional Neural Network for Pansharpening with Spatial Structure Enhancement Operator

A New Context-Aware Details Injection Fidelity With Adaptive Coefficients Estimation for Variational Pansharpening

PAN-Guided Band-Aware Multi-Spectral Feature Enhancement for Pan-Sharpening

An Unsupervised Generative Adversarial Network Based on U-Shaped Structure for Pansharpening

A Variational Pan-Sharpening with Local Gradient Constraints.

A Survey of Pansharpening Methods with A New Band-Decoupled Variational Model

Pansharpening of WorldView-2 Data via Graph Regularized Sparse Coding and Adaptive Coupled Dictionary

LRTCFPan: Low-Rank Tensor Completion Based Framework for Pansharpening

Multiscale Spatial-Frequency Domain Dynamic Pansharpening of Remote Sensing Images Integrated With Wavelet Transform

PanAdapter: Two-Stage Fine-Tuning with Spatial-Spectral Priors Injecting for Pansharpening

Pansharpening via semi-framelet-guided sparse reconstruction

Detail Information Prior Net for Remote Sensing Image Pansharpening

Sparse Representation over Shared Coefficients in Multispectral Pansharpening

A Joint Optimization Based Pansharpening via Subpixel-Shift Decomposition

Memory-augmented Deep Conditional Unfolding Network for Pansharpening