Abstract:As AI workloads increase in scope, generalization capability becomes challenging for small task-specific models and their demand for large amounts of labeled training samples increases. On the contrary, Foundation Models (FMs) are trained with internet-scale unlabeled data via self-supervised learning and have been shown to adapt to various tasks with minimal fine-tuning. Although large FMs have demonstrated significant impact in natural language processing and computer vision, efforts toward FMs for geospatial applications have been restricted to smaller size models, as pretraining larger models requires very large computing resources equipped with state-of-the-art hardware accelerators. Current satellite constellations collect 100+TBs of data a day, resulting in images that are billions of pixels and multimodal in nature. Such geospatial data poses unique challenges opening up new opportunities to develop FMs. We investigate billion scale FMs and HPC training profiles for geospatial applications by pretraining on publicly available data. We studied from end-to-end the performance and impact in the solution by scaling the model size. Our larger 3B parameter size model achieves up to 30% improvement in top1 scene classification accuracy when comparing a 100M parameter model. Moreover, we detail performance experiments on the Frontier supercomputer, America's first exascale system, where we study different model and data parallel approaches using PyTorch's Fully Sharded Data Parallel library. Specifically, we study variants of the Vision Transformer architecture (ViT), conducting performance analysis for ViT models with size up to 15B parameters. By discussing throughput and performance bottlenecks under different parallelism configurations, we offer insights on how to leverage such leadership-class HPC resources when developing large models for geospatial imagery applications.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on the following aspects: 1. **Limitations of task - specific models**: Currently, deep - learning models used for Earth observation data analysis are usually designed for specific tasks, and their generalization ability is limited. In particular, they perform poorly when dealing with unseen data (i.e., out - of - distribution data). 2. **Requirement for a large amount of labeled data**: These models need a large number of manually labeled data samples for training, which not only increases the cost of model development but also limits the application scope of the models. 3. **Development of large - scale foundation models in geospatial applications**: Although large - scale foundation models (FMs) have made remarkable progress in fields such as natural language processing and computer vision, in geospatial applications, relevant research is mostly limited to smaller - scale models because pre - training large models requires extremely large computing resources. To solve the above problems, the authors have made the following explorations: - **Pre - training of large - scale models**: By using publicly available datasets to pre - train visual transformer (ViT) models with billions of parameters, the performance of different models and data parallel strategies on the Frontier supercomputer is explored. - **Performance evaluation**: Through linear probing experiments on ViT models of different sizes, the significant advantages of large - scale models in remote - sensing image classification tasks are demonstrated, with a maximum 30% improvement in top - level scene classification accuracy. - **Distributed training methods**: The methods for large - scale model training using the fully sharded data parallel (FSDP) library of PyTorch on the Frontier supercomputer are discussed in detail, including performance optimization and communication cost analysis under different sharding strategies. Through these explorations, the paper aims to provide practical guidelines and technical support for the development of large - scale foundation models in the field of geospatial data analysis and promote the further development of this field.

Pretraining Billion-scale Geospatial Foundational Models on Frontier

OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

Optimizing Distributed Training on Frontier for Large Language Models

Foundation Models for Generalist Geospatial Artificial Intelligence

Training Large-scale Foundation Models on Emerging AI Chips

A Billion-scale Foundation Model for Remote Sensing Images

Towards Geospatial Foundation Models via Continual Pretraining

On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence

On the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)

Decentralized Training of Foundation Models in Heterogeneous Environments

When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System

An Efficient 2D Method for Training Super-Large Deep Learning Models

Enabling Foundation Models: A Distributed Collaboration Framework Based on Graph Federated Learning

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

Parallelizing Training of Deep Generative Models on Massive Scientific Datasets

Comparative Study of Large Language Model Architectures on Frontier

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

Evaluation of pre-training large language models on leadership-class supercomputers

Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations

Efficient Large-Scale Language Model Training on GPU Clusters