Abstract:Self-supervised pretraining on large-scale satellite data has raised great interest in building Earth observation (EO) foundation models. However, many important resources beyond pure satellite imagery, such as land-cover-land-use products that provide free global semantic information, as well as vision foundation models that hold strong knowledge of the natural world, are not widely studied. In this work, we show these free additional resources not only help resolve common contrastive learning bottlenecks, but also significantly boost the efficiency and effectiveness of EO pretraining. Specifically, we first propose soft contrastive learning that optimizes cross-scene soft similarity based on land-cover-generated multi-label supervision, naturally solving the issue of multiple positive samples and too strict positive matching in complex scenes. Second, we revisit and explore cross-domain continual pretraining for both multispectral and SAR imagery, building efficient EO foundation models from strongest vision models such as DINOv2. Adapting simple weight-initialization and Siamese masking strategies into our soft contrastive learning framework, we demonstrate impressive continual pretraining performance even when the input modalities are not aligned. Without prohibitive training, we produce multispectral and SAR foundation models that achieve significantly better results in 10 out of 11 downstream tasks than most existing SOTA models. For example, our ResNet50/ViT-S achieve 84.8/85.0 linear probing mAP scores on BigEarthNet-10\% which are better than most existing ViT-L models; under the same setting, our ViT-B sets a new record of 86.8 in multispectral, and 82.5 in SAR, the latter even better than many multispectral models. Dataset and models are available at \url{<a class="link-external link-https" href="https://github.com/zhu-xlab/softcon" rel="external noopener nofollow">this https URL</a>}.

SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation

SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation [Software and Data Sets]

A generic self-supervised learning (SSL) framework for representation learning from spectra-spatial feature of unlabeled remote sensing imagery

Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining

SSL4EO-L: Datasets and Foundation Models for Landsat Imagery

Remote Sensing Image Scene Classification with Self-Supervised Paradigm under Limited Labeled Samples

Self-Supervised Learning on Small In-Domain Datasets Can Overcome Supervised Learning in Remote Sensing

Multilabel-Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining

Self-Supervised Learning in Remote Sensing: A review

Self-supervised Audiovisual Representation Learning for Remote Sensing Data

OmniSat: Self-Supervised Modality Fusion for Earth Observation

Self-Supervised Learning on MeerKAT Wide-Field Continuum Images

SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding

SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery

Open Self-Supervised Features for Remote-Sensing Image Scene Classification Using Very Few Samples

CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding

Semantic-Aware Dense Representation Learning for Remote Sensing Image Change Detection

Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data

SpectralEarth: Training Hyperspectral Foundation Models at Scale

SSL-SoilNet: A Hybrid Transformer-based Framework with Self-Supervised Learning for Large-scale Soil Organic Carbon Prediction