Remote Sensing Image Fusion Based on Two-stream Fusion Network

Xiangyu Liu,Qingjie Liu,Yunhong Wang

DOI: https://doi.org/10.48550/arXiv.1711.02549

2018-01-26

Abstract:Remote sensing image fusion (also known as pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. Inspired by the astounding achievements of convolutional neural networks (CNNs) in a variety of computer vision tasks, in this paper, we propose a two-stream fusion network (TFNet) to address the problem of pan-sharpening. Unlike previous CNN based methods that consider pan-sharpening as a super resolution problem and perform pan-sharpening in pixel level, the proposed TFNet aims to fuse PAN and MS images in feature level and reconstruct the pan-sharpened image from the fused features. The TFNet mainly consists of three parts. The first part is comprised of two networks extracting features from PAN and MS images, respectively. The subsequent network fuses them together to form compact features that represent both spatial and spectral information of PAN and MS images, simultaneously. Finally, the desired high spatial resolution MS image is recovered from the fused features through an image reconstruction network. Experiments on Quickbird and \mbox{GaoFen-1} satellite images demonstrate that the proposed TFNet can fuse PAN and MS images, effectively, and produce pan-sharpened images competitive with even superior to state of the arts.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the fusion problem of multispectral images and panchromatic images in remote - sensing image fusion, that is, how to generate a multispectral image with high spatial resolution from a single - band panchromatic (PAN) image with high spatial resolution and a multispectral (MS) image with low spatial resolution. This process is also known as "pan - sharpening". Specifically, the paper proposes a method based on the Two - stream Fusion Network (TFNet) to solve this problem. This method aims to fuse PAN and MS images at the feature level rather than at the pixel level, and restores a high - resolution multispectral image through a reconstruction network.

The main contributions of the paper include:
1. Proposing a two - stream convolutional neural network architecture to solve the pan - sharpening problem and complete the fusion at the feature level.
2. Using the \( \ell_1 \) loss function instead of the widely - used \( \ell_2 \) loss function to optimize the network, achieving better results.
3. Exploring the residual learning technique to further improve the performance of the two - stream fusion network, proving the effectiveness of residual learning in the pan - sharpening problem.

Through these innovations, the method proposed in the paper can effectively fuse PAN and MS images when handling remote - sensing image fusion tasks, and the generated pan - sharpened images are superior or at least comparable to existing methods in terms of quality and detail.