Abstract:As AI workloads increase in scope, generalization capability becomes challenging for small task-specific models and their demand for large amounts of labeled training samples increases. On the contrary, Foundation Models (FMs) are trained with internet-scale unlabeled data via self-supervised learning and have been shown to adapt to various tasks with minimal fine-tuning. Although large FMs have demonstrated significant impact in natural language processing and computer vision, efforts toward FMs for geospatial applications have been restricted to smaller size models, as pretraining larger models requires very large computing resources equipped with state-of-the-art hardware accelerators. Current satellite constellations collect 100+TBs of data a day, resulting in images that are billions of pixels and multimodal in nature. Such geospatial data poses unique challenges opening up new opportunities to develop FMs. We investigate billion scale FMs and HPC training profiles for geospatial applications by pretraining on publicly available data. We studied from end-to-end the performance and impact in the solution by scaling the model size. Our larger 3B parameter size model achieves up to 30% improvement in top1 scene classification accuracy when comparing a 100M parameter model. Moreover, we detail performance experiments on the Frontier supercomputer, America's first exascale system, where we study different model and data parallel approaches using PyTorch's Fully Sharded Data Parallel library. Specifically, we study variants of the Vision Transformer architecture (ViT), conducting performance analysis for ViT models with size up to 15B parameters. By discussing throughput and performance bottlenecks under different parallelism configurations, we offer insights on how to leverage such leadership-class HPC resources when developing large models for geospatial imagery applications.

BaGuaLu

BaGuaLu: targeting brain scale pretrained models with over 37 million cores

P F ] 1 3 A ug 2 01 9 HPC AI 500 : A Benchmark Suite for HPC AI Systems

HPC AI500: A Benchmark Suite for HPC AI Systems

DaDianNao: A Machine-Learning Supercomputer

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

HPC-GPT: Integrating Large Language Model for High-Performance Computing

GAIA-Universe: Everything is Super-Netify

Pretraining Billion-scale Geospatial Foundational Models on Frontier

Chiplet Cloud: Building AI Supercomputers for Serving Large Generative Language Models

The Solution for the AIGC Inference Performance Optimization Competition

HetHub: A Heterogeneous Distributed Hybrid Training System for Large-Scale Models

SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts

HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models

Development of GPU Enhanced AI Education Platforms for K-12 Schools

Computing in the Era of Large Generative Models: From Cloud-Native to AI-Native