Abstract:Most of the existing deep learning-based hyperspectral image (HSI) classification algorithms are based on supervised learning, where a large number of annotated labels with high acquisition cost are required. Self-supervised learning (SSL) methods can learn abundant representations using a large amount of unlabeled data, thereby reducing the reliability of labels. In particular, SSL based on masked image modeling (MIM) can extract fine-grained features, which is well suited for HSI classification as a pixel-level interpretation task. However, MIM has scarcely been investigated in the HSI classification. Current algorithms lack a comprehensive consideration of the multiscale spectral–spatial characteristics of HSI when constructing the pretraining task, and there exists high computational cost and redundancy when applied to large-scale HSIs. Therefore, this article develops an SSL framework based on spectral–spatial hierarchical masked modeling (S2HM2) for large-scale HSI classification. Considering the spectral–spatial characteristics of HSI, 3-D masking strategy and spectral–spatial consistency loss are proposed to construct the MIM task. To fully exploit features at each scale, hierarchical 3-D feature pyramid network (3D-FPN) is designed as decoder for both pretext and downstream tasks in a “pixel-to-pixel” manner. In addition, multiscale masked feature modeling (MS-MFM) task is proposed to further facilitate the multiscale feature learning. The SSL pretraining is guided by both MIM and MS-MFM. The experimental results on two large-scale hyperspectral datasets, i.e., WHU-OHS and WHU-H2SR, demonstrate the superiority of the proposed method. Furthermore, transfer learning experiments are conducted on a variety of hyperspectral datasets, where classification accuracies are boosted in most of the scenarios. The source code will be made available at https://github.com/tulilin/S2HM2.

LFSMIM: A Low-Frequency Spectral Masked Image Modeling Method for Hyperspectral Image Classification

S2HM2: A Spectral–Spatial Hierarchical Masked Modeling Framework for Self-Supervised Feature Learning and Classification of Large-Scale Hyperspectral Images

Remote Sensing Scene Classification with Masked Image Modeling (MIM)

SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification

Self-Supervised Feature Learning Based on Spectral Masking for Hyperspectral Image Classification

Symmetric masking strategy enhances the performance of Masked Image Modeling

Few-Shot Learning With Mutual Information Enhancement for Hyperspectral Image Classification

Boosting Spatial-Spectral Masked Auto-Encoder Through Mining Redundant Spectra for HSI-SAR/LiDAR Classification

MSMT-LCL: Multiscale Spatial-Spectral Masked Transformer With Local Contrastive Learning for Hyperspectral Image Classification

SimMIM: A Simple Framework for Masked Image Modeling

Improving Pixel-based MIM by Reducing Wasted Modeling Capability

Exploring the Coordination of Frequency and Attention in Masked Image Modeling

Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentation

HSIMamba: Hyperpsectral Imaging Efficient Feature Learning with Bidirectional State Space for Classification

CtxMIM: Context-Enhanced Masked Image Modeling for Remote Sensing Image Understanding

PixMIM: Rethinking Pixel Reconstruction in Masked Image Modeling

Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learning

Hyperspectral image classification using spectral-spatial LSTMs

Scaling Efficient Masked Image Modeling on Large Remote Sensing Dataset

Interactive Masked Image Modeling for Multimodal Object Detection in Remote Sensing