Abstract:Intelligent interpretation of remote sensing images using deep learning is heavily reliant on large datasets, and models trained in one domain often struggle with crossdomain application. Pretraining the backbone network via masked image modeling can effectively diminish this reliance on extensive sample data, thereby reducing crossdomain transfer obstacles. However, current masked image models typically employ a pure Transformer architecture, which may not fully capitalize on low-level features. To address these issues, this article proposes masked feature modeling (MFM), a methodology for the generative self-supervised learning of high-resolution remote sensing images that combines convolutional neural network (CNN) and Transformer architectures. This methodology has several advantages: 1) The hybrid CNN + Transformer architecture not only retains the advantages of the local feature representation of the CNN architecture but also has the full-text information modeling capabilities of the Transformer architecture; 2) the feature extraction network outputs multiscale features, and it is easier to add upsampling and a skip connection to improve the accuracy of the downstream dense prediction task; and 3) the pretrained MFM can be applied to various downstream tasks through fine-tuning with limited samples. The publicly available WHU and Massachusetts Building Datasets are used to verify the effectiveness of the proposed method. Extensive experiments involving main properties of the MFM for generative self-supervised learning, fine-tuning the MFM on the downstream semantic segmentation task, and comparisons with the other state-of-the-art generative self-supervised learning algorithms show that, through the combined advantages of the CNN and Transformer architectures, the proposed method has better feature extraction capability and higher accuracy on downstream tasks such as semantic segmentation.

Generative ConvNet Foundation Model With Sparse Modeling and Low-Frequency Reconstruction for Remote Sensing Image Interpretation

SpectralGPT: Spectral Remote Sensing Foundation Model

When are Foundation Models Effective? Understanding the Suitability for Pixel-Level Classification Using Multispectral Imagery

A Billion-scale Foundation Model for Remote Sensing Images

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

Large Kernel Sparse ConvNet Weighted by Multi-Frequency Attention for Remote Sensing Scene Understanding

SatVision-TOA: A Geospatial Foundation Model for Coarse-Resolution All-Sky Remote Sensing Imagery

Masked Feature Modeling for Generative Self-Supervised Representation Learning of High-Resolution Remote Sensing Images

AI Foundation Models in Remote Sensing: A Survey

Single Remote Sensing Image Super-Resolution Via a Generative Adversarial Network with Stratified Dense Sampling and Chain Training

One for All: Toward Unified Foundation Models for Earth Vision

Towards a Knowledge guided Multimodal Foundation Model for Spatio-Temporal Remote Sensing Applications

Brain-Inspired Remote Sensing Foundation Models and Open Problems: A Comprehensive Survey

Foundation Models for Generalist Geospatial Artificial Intelligence

Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model

SpectralGPT: Spectral Foundation Model

RingMo: A Remote Sensing Foundation Model with Masked Image Modeling

MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining

Foundation Model-Based Multimodal Remote Sensing Data Classification

A Unified Generative Adversarial Network With Convolution and Transformer for Remote Sensing Image Fusion