Abstract:Pan-sharpening is an important technique for remote sensing imaging systems to obtain high resolution multispectral images. Recently, deep learning has become the most popular tool for pan-sharpening. This paper develops a model-based deep pan-sharpening approach. Specifically, two optimization problems regularized by the deep prior are formulated, and they are separately responsible for the generative models for panchromatic images and low resolution multispectral images. Then, the two problems are solved by a gradient projection algorithm, and the iterative steps are generalized into two network blocks. By alternatively stacking the two blocks, a novel network, called gradient projection based pan-sharpening neural network, is constructed. The experimental results on different kinds of satellite datasets demonstrate that the new network outperforms state-of-the-art methods both visually and quantitatively. The codes are available at <a class="link-external link-https" href="https://github.com/xsxjtu/GPPNN" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to effectively fuse the information of low - resolution multispectral images (LRMS) and panchromatic images (PAN) in the generation of high - resolution multispectral images (HRMS). Specifically, the author proposes a model - driven deep - learning method, namely the Gradient Projection Pansharpening Neural Network (GPPNN), to improve the quality of image fusion. Through this method, multispectral images with high spatial resolution and high quality can be generated, thus surpassing the existing state - of - the - art methods in both visual effects and quantitative evaluations.
### Background of the Paper
Pansharpening technology is an important remote - sensing imaging technology for generating high - resolution multispectral images (HRMS) from low - resolution multispectral images (LRMS) and high - resolution panchromatic images (PAN). In recent years, deep - learning methods have made significant progress in the field of pansharpening, but most methods lack the interpretability of the observation model and require a large number of training samples.
### Contributions of the Paper
1. **Model Construction**: The paper proposes two optimization problems for generating the generation models of panchromatic images and low - resolution multispectral images respectively. These two optimization problems are regularized by deep priors and solved using the gradient projection algorithm.
2. **Network Design**: Inspired by the algorithm unrolling technique, the iterative steps of the optimization problems are transformed into two neural network modules (MS Block and PAN Block), and the GPPNN is constructed by alternately stacking these two modules.
3. **Performance Verification**: The experimental results show that the GPPNN outperforms the existing state - of - the - art methods on multiple satellite datasets, both in terms of visual effects and quantitative evaluation metrics.
### Technical Details
- **Model Formulation**: It is assumed that the LRMS image is obtained by blurring and down - sampling the HRMS image, and the PAN image is the spectral response result of the HRMS image. Based on this assumption, two optimization problems are constructed:
\[
\min_H \frac{1}{2} \|L - DKH\|_2^2+\lambda h_l(H)
\]
\[
\min_H \frac{1}{2} \|P - HS\|_2^2+\lambda h_p(H)
\]
where \(L\) is the LRMS image, \(P\) is the PAN image, \(H\) is the HRMS image, \(D\) is the down - sampling matrix, \(K\) is the low - pass convolution matrix, \(S\) is the spectral response function, and \(h_l\) and \(h_p\) are deep priors.
- **MS Block and PAN Block**: These two modules are respectively responsible for processing the information of the LRMS image and the PAN image and updating the estimated value of the HRMS image by the gradient projection method. Each module contains four main steps: generating a predicted image, calculating the residual, up - sampling the residual, and updating the HRMS image.
- **GPPNN Structure**: The GPPNN is constructed by alternately stacking the MS Block and the PAN Block, and the initial value is obtained from the LRMS image by bicubic interpolation. The output of the network is the final HRMS image.
### Experimental Results
- **Dataset**: The experiments are carried out on three satellite datasets, Landsat8, QuickBird, and GaoFen2.
- **Evaluation Metrics**: Four metrics, PSNR, SSIM, ERGAS, and SAM, are used for evaluation.
- **Performance Comparison**: The GPPNN performs excellently on all datasets, especially significantly outperforming other methods in terms of PSNR and SSIM metrics.
### Conclusion
The paper proposes a model - driven deep - learning method, GPPNN. By combining the observation model and deep priors, it effectively solves the pansharpening problem. The experimental results verify the effectiveness and superiority of this method. Future work will focus on improving the generalization ability of GPPNN so that it can be applied to data from different satellites.