Abstract:As AI workloads increase in scope, generalization capability becomes challenging for small task-specific models and their demand for large amounts of labeled training samples increases. On the contrary, Foundation Models (FMs) are trained with internet-scale unlabeled data via self-supervised learning and have been shown to adapt to various tasks with minimal fine-tuning. Although large FMs have demonstrated significant impact in natural language processing and computer vision, efforts toward FMs for geospatial applications have been restricted to smaller size models, as pretraining larger models requires very large computing resources equipped with state-of-the-art hardware accelerators. Current satellite constellations collect 100+TBs of data a day, resulting in images that are billions of pixels and multimodal in nature. Such geospatial data poses unique challenges opening up new opportunities to develop FMs. We investigate billion scale FMs and HPC training profiles for geospatial applications by pretraining on publicly available data. We studied from end-to-end the performance and impact in the solution by scaling the model size. Our larger 3B parameter size model achieves up to 30% improvement in top1 scene classification accuracy when comparing a 100M parameter model. Moreover, we detail performance experiments on the Frontier supercomputer, America's first exascale system, where we study different model and data parallel approaches using PyTorch's Fully Sharded Data Parallel library. Specifically, we study variants of the Vision Transformer architecture (ViT), conducting performance analysis for ViT models with size up to 15B parameters. By discussing throughput and performance bottlenecks under different parallelism configurations, we offer insights on how to leverage such leadership-class HPC resources when developing large models for geospatial imagery applications.

BaGuaLu: targeting brain scale pretrained models with over 37 million cores

BaGuaLu

DaDianNao: A Machine-Learning Supercomputer

PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

SuperGCN: General and Scalable Framework for GCN Training on CPU-powered Supercomputers

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining

An Efficient 2D Method for Training Super-Large Deep Learning Models

Parallelizing Training of Deep Generative Models on Massive Scientific Datasets

Evaluation of pre-training large language models on leadership-class supercomputers

21296 Exponentially Complex Quantum Many-Body Simulation via Scalable Deep Learning Method

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models

Elixir: Train a Large Language Model on a Small GPU Cluster

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Pretraining Billion-scale Geospatial Foundational Models on Frontier

LuWu: An End-to-End In-Network Out-of-Core Optimizer for 100B-Scale Model-in-Network Data-Parallel Training on Distributed GPUs

$2^{1296}$ Exponentially Complex Quantum Many-Body Simulation via Scalable Deep Learning Method