Attention based Dual-Branch Complex Feature Fusion Network for Hyperspectral Image Classification

Mohammed Q. Alkhatib,Mina Al-Saad,Nour Aburaed,M. Sami Zitouni,Hussain Al Ahmad

2023-11-03

Abstract:This research work presents a novel dual-branch model for hyperspectral image classification that combines two streams: one for processing standard hyperspectral patches using Real-Valued Neural Network (RVNN) and the other for processing their corresponding Fourier transforms using Complex-Valued Neural Network (CVNN). The proposed model is evaluated on the Pavia University and Salinas datasets. Results show that the proposed model outperforms state-of-the-art methods in terms of overall accuracy, average accuracy, and Kappa. Through the incorporation of Fourier transforms in the second stream, the model is able to extract frequency information, which complements the spatial information extracted by the first stream. The combination of these two streams improves the overall performance of the model. Furthermore, to enhance the model performance, the Squeeze and Excitation (SE) mechanism has been utilized. Experimental evidence show that SE block improves the models overall accuracy by almost 1\%.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges in hyperspectral image classification. Specifically, the author proposes a new dual - branch model, aiming to improve the accuracy of hyperspectral image classification by combining two streams:

1. **The first stream**: Use the real - valued neural network (RVNN) to process standard hyperspectral image patches and extract spatial and spectral features.
2. **The second stream**: Use the complex - valued neural network (CVNN) to process the Fourier transform of these image patches and extract frequency information.

By combining the information of these two streams, the model can utilize both spatial information and frequency information simultaneously, thus improving the classification performance. In addition, in order to further enhance the performance of the model, the author also introduces the Squeeze and Excitation (SE) mechanism to strengthen the dependencies between channels and improve the feature representation ability.

The main contributions of the paper are:
- Proposing a novel dual - branch model that combines real - valued and complex - valued neural networks.
- Utilizing the Fourier transform to extract frequency information, supplementing spatial information.
- Introducing the SE mechanism to further improve the overall performance of the model.

The experimental results show that this model is superior to existing methods in terms of overall accuracy, average accuracy, and Kappa coefficient on the Pavia University and Salinas datasets.