Joint Self-Attention Transformer for Multispectral and Hyperspectral Image Fusion
Li Miaoyu,Fu Ying
DOI: https://doi.org/10.11834/jig.220954
2023-01-01
Journal of Image and Graphics
Abstract:Objective Hyperspectral image(HSI)contains rich spectral information and has advantages over multispectral image(MSI)in accurately distinguishing different types of materials.Therefore,HSI has been widely used in many com-puter vision tasks,including vegetation detection,face recognition,and feature segmentation.However,due to the limita-tions in hardware equipment and the acquisition environment,an inevitable trade-off arises between spatial resolution and spectral resolution.Thus,HSIs under real scenes often have low spatial resolution,which negatively affects the perfor-mance of subsequent vision tasks.By fusing the low-resolution HSI(LR-HSI)with a high-resolution MSI(HR-MSI)under the same scene using the HSI super-resolution algorithm,the spatial resolution of HSIs can be effectively improved.Exist-ing HSI fusion algorithms can be roughly classified into traditional-model-based and deep-learning-based methods.Traditional-model-based fusion methods employ various handcrafted shallow priors(e.g.,matrix/tensor factorization,total variation,and low rank)to utilize the intrinsic statistics of observed spectral images.However,these methods lack general-ization ability to complex real scenarios and consume much time in iteratively optimizing the designed prior.Meanwhile,deep-learning-based fusion methods can automatically learn the prior knowledge from large-scale datasets.Although these methods often achieve better fusion results compared with traditional-model-based fusion methods,they do not jointly explore the inner self-similarity of multi-source spectral images,where the LR-HSI shows high correlation in the spectral dimension and the HR-MSI shows spatial similarities in texture and edges.In addition,the weights of these convolution-based networks are learned during training but are fixed during testing,hence limiting the potential adaptability of net-works.To effectively exploit the inner spatial and spectral similarity of spectral images,we propose an MSI and HSI fusion network with a joint self-attention fusion module and Transformer.Method Given that LR-HSI has reliable information in the spectral dimension,the critical task of the HSI fusion method is to fill the missing texture details in the spatial dimen-sion without losing discriminable spectral information.Given the LR-HSI and its matching HR-MSI,our proposed method fuses these two spectral images to obtain the desired HR-HSI in three steps.First,the similarity information of LR-HSI and HR-MSI is extracted by the joint self-attention module.Specifically,the spectral similarity features from LR-HSI are extracted by the channel attention module,and the spatial similarity features from HR-MSI are extracted by the spatial attention module.The obtained similarity features are then used to guide the fusion process.Second,to achieve a deep rep-resentation and explore the long-range dependencies of the fusion features,the preliminary fusion features are fed into the deep Transformer network,which comprises a shift window attention module,LayerNorm,and multilayer perceptron.The convolution layer and skip connection are also included in the proposed Transformer fusion network to further enhance the model flexibility.Third,the fusion features from Transformer are mapped to the desired high-resolution HSI.The overall network is implemented by the Pytorch framework and trained in an end-to-end manner.To generate training data,the training images are cropped to the size of 96 × 96 × 31,resulting in approximately 8 000 training patches that are smoothed by a Gaussian blur kernel and spatially down-sampled to obtain LR-HSI.The MSI images are generated by the spectral response function of a Nikon D700 camera.Result We compare our method with seven state-of-the-art fusion methods,including one traditional-model-based method and six deep-leaming-based methods.The peak-signal-to-noise ratio(PSNR),structural similarity index measure(SSIM),erreur relative globale adimensionnelle de Synthèse(ERGAS),and spectral angle mapper(SAM)are utilized as quantitative metrics in evaluating the performance of these fusion methods.To verify the effectiveness of the proposed model,we perform experiments on two widely used HSI datasets,namely,the CAVE and Harvard datasets.For the CAVE dataset,the first 20 images are selected for training,and the last 12 images are used for testing.Similarly,for the Harvard dataset,the first 30 images are selected for training,and the last 20 images are used for testing.Experimental results under different scale factors show that the proposed method achieves better fusion results in terms of quantitative metrics and visual effects compared to the other state-of-the-art methods.Under a scale fac-tor of 8,the PSNR,SAM,and ERGAS of the proposed method is improved by 0.5 dB,0.13,and 0.2,respectively,com-pared to EDBIN,which is the second best-performing method on the CAVE dataset.Under a scale factor 16,the PSNR of the proposed method is improved by at least 0.4 dB compared to the other methods on the Harvard dataset.The visual results show that our proposed method outperforms the other methods in recovering both fine-grained spatial textures and spectral details.The ablation study also proves that the employed Transformer fusion network significantly improves the fusion process.Conclusion In this paper,we propose a Transformer-based MSI and HSI fusion network with a joint self-attention fusion module,which can effectively utilize the spectral similarity of LR-HSI and the spatial similarity of HR-MSI to guide the fusion process through a 2D attention mechanism.The preliminary fusion results pass through the residual Transformer network to obtain a deep feature representation and to reconstruct the desired HR-HSI.Qualitative and quanti-tative experiments show that the proposed method has better spectral fidelity and spatial resolution compared to the state-of-the-art HSI fusion methods.