Johannes Jakubik,Sujit Roy,C. E. Phillips,Paolo Fraccaro,Denys Godwin,Bianca Zadrozny,Daniela Szwarcman,Carlos Gomes,Gabby Nyirjesy,Blair Edwards,Daiki Kimura,Naomi Simumba,Linsong Chu,S. Karthik Mukkavilli,Devyani Lambhate,Kamal Das,Ranjini Bangalore,Dario Oliveira,Michal Muszynski,Kumar Ankur,Muthukumaran Ramasubramanian,Iksha Gurung,Sam Khallaghi,Hanxi,Michael Cecil,Maryam Ahmadi,Fatemeh Kordi,Hamed Alemohammad,Manil Maskey,Raghu Ganti,Kommy Weldemariam,Rahul Ramachandran

Abstract:Significant progress in the development of highly adaptable and reusable Artificial Intelligence (AI) models is expected to have a significant impact on Earth science and remote sensing. Foundation models are pre-trained on large unlabeled datasets through self-supervision, and then fine-tuned for various downstream tasks with small labeled datasets. This paper introduces a first-of-a-kind framework for the efficient pre-training and fine-tuning of foundational models on extensive geospatial data. We have utilized this framework to create Prithvi, a transformer-based geospatial foundational model pre-trained on more than 1TB of multispectral satellite imagery from the Harmonized Landsat-Sentinel 2 (HLS) dataset. Our study demonstrates the efficacy of our framework in successfully fine-tuning Prithvi to a range of Earth observation tasks that have not been tackled by previous work on foundation models involving multi-temporal cloud gap imputation, flood mapping, wildfire scar segmentation, and multi-temporal crop segmentation. Our experiments show that the pre-trained model accelerates the fine-tuning process compared to leveraging randomly initialized weights. In addition, pre-trained Prithvi compares well against the state-of-the-art, e.g., outperforming a conditional GAN model in multi-temporal cloud imputation by up to 5pp (or 5.7%) in the structural similarity index. Finally, due to the limited availability of labeled data in the field of Earth observation, we gradually reduce the quantity of available labeled data for refining the model to evaluate data efficiency and demonstrate that data can be decreased significantly without affecting the model's accuracy. The pre-trained 100 million parameter model and corresponding fine-tuning workflows have been released publicly as open source contributions to the global Earth sciences community through Hugging Face.

Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation

One for All: Toward Unified Foundation Models for Earth Vision

Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark

Cross-Domain Foundation Model Adaptation: Pioneering Computer Vision Models for Geophysical Data Analysis

Generative ConvNet Foundation Model With Sparse Modeling and Low-Frequency Reconstruction for Remote Sensing Image Interpretation

Foundation Models for Remote Sensing and Earth Observation: A Survey

Foundation Models for Generalist Geospatial Artificial Intelligence

Earthfarsser: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model

EarthPT: a time series foundation model for Earth Observation

Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model

On the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)

On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence

Foundation Model-Based Multimodal Remote Sensing Data Classification

Towards a Knowledge guided Multimodal Foundation Model for Spatio-Temporal Remote Sensing Applications

A Foundation Model for the Earth System

Foundation model of neural activity predicts response to new stimulus types and anatomy

Seismic Foundation Model (SFM): a new generation deep learning model in geophysics

Brain-Inspired Remote Sensing Foundation Models and Open Problems: A Comprehensive Survey

RS-DFM: A Remote Sensing Distributed Foundation Model for Diverse Downstream Tasks

Toward Foundation Models for Earth Monitoring: Generalizable Deep Learning Models for Natural Hazard Segmentation