Abstract:libEnsemble is a Python-based toolkit for running dynamic ensembles, developed as part of the DOE Exascale Computing Project. The toolkit utilizes a unique generator--simulator--allocator paradigm, where generators produce input for simulators, simulators evaluate those inputs, and allocators decide whether and when a simulator or generator should be called. The generator steers the ensemble based on simulation results. Generators may, for example, apply methods for numerical optimization, machine learning, or statistical calibration. libEnsemble communicates between a manager and workers. We overview the unique characteristics of libEnsemble as well as current and potential interoperability with other packages in the workflow ecosystem. We highlight libEnsemble's dynamic resource features: libEnsemble can detect system resources, such as available nodes, cores, and GPUs, and assign these in a portable way. These features allow users to specify the number of processors and GPUs required for each simulation; and resources will be automatically assigned on a wide range of systems, including Frontier, Aurora, and Perlmutter. Such ensembles can include multiple simulation types, some using GPUs and others using only CPUs, sharing nodes for maximum efficiency. We also describe the benefits of libEnsemble's generator--simulator coupling, which easily exposes to the user the ability to cancel, and portably kill, running simulations based on models that are updated with intermediate simulation output. We demonstrate libEnsemble's capabilities, scalability, and scientific impact via a Gaussian process surrogate training problem for the longitudinal density profile at the exit of a plasma accelerator stage. The study uses gpCAM for the surrogate model and employs either Wake-T or WarpX simulations, highlighting efficient use of resources that can easily extend to exascale.

Scaling on Frontier: Uncertainty Quantification Workflow Applications using ExaWorks to Enable Full System Utilization

Exascale Workflow Applications and Middleware: An ExaWorks Retrospective

ExaWorks Software Development Kit: A Robust and Scalable Collection of Interoperable Workflow Technologies

Experiences Readying Applications for Exascale

Early experiences on the OLCF Frontier system with AthenaPK and Parthenon‐Hydro

Portable, heterogeneous ensemble workflows at scale using libEnsemble

Asynchronous-Many-Task Systems: Challenges and Opportunities -- Scaling an AMR Astrophysics Code on Exascale machines using Kokkos and HPX

Exascale Computational Fluid Dynamics in Heterogeneous Systems

Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance Tracking: the CyberShake Example

Towards Exascale for Wind Energy Simulations

RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms

Creating Continuous Integration Infrastructure for Software Development on DOE HPC Systems

Exploring the Frontiers of Energy Efficiency using Power Management at System Scale

Deploying Optimized Scientific and Engineering Applications on Exascale Systems

Adaptive Computing for Scale-up Problems

ytopt: Autotuning Scientific Applications for Energy Efficiency at Large Scales

Leveraging DevOps for Scientific Computing

Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

Scalable Delivery of Scalable Libraries and Tools: How ECP Delivered a Software Ecosystem for Exascale and Beyond

Simulation Factory: Taming Application Configuration and Workflow on High-End Resources

Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments