Abstract:ExaGRyPE describes a suite of solvers for numerical relativity, based upon ExaHyPE 2, the second generation of our Exascale Hyperbolic PDE Engine. The presented generation of ExaGRyPE solves the Einstein equations in the CCZ4 formulation under a 3+1 foliation and focuses on black hole spacetimes. It employs a block-structured Cartesian grid carrying a higher-order order Finite Difference scheme with adaptive mesh refinement, it facilitates massive parallelism combining message passing, domain decomposition and task parallelism, and it supports the injection of particles into the grid as data probes or tracers. We introduce the ExaGRyPE-specific building blocks within ExaHyPE 2, and discuss its software architecture and compute-n-feel: For this, we formalize the creation of any specific simulation with ExaGRyPE as a sequence of lowering operations, where abstract logical tasks are successively broken into finer tasks until we obtain an abstraction level that can be mapped onto a C++ executable. The overall program logic is fully specified via a domain-specific Python interface, we map this logic onto a more detailed set of numerical tasks, subsequently lower this representation onto technical tasks that the underlying ExaHyPE engine uses to parallelize the application, before eventually the technical tasks are mapped onto task graphs including the actual PDE term evaluations, initial conditions, boundary conditions, and so forth. We end up with a rigorous separation of concerns which shields ExaGRyPE users from technical details and hence simplifies the development of novel physical models. We present the simulations and data for the gauge wave, static single black holes and rotating binary black hole systems, demonstrating that the code base is mature and usable. However, we also uncover domain-specific numerical challenges that need further study by the community in future work.

Performance-Portable Numerical Relativity with AthenaK

Performance-Portable Binary Neutron Star Mergers with AthenaK

AthenaK: A Performance-Portable Version of the Athena++ AMR Framework

GR-Athena++: Puncture Evolutions on Vertex-centered Oct-tree Adaptive Mesh Refinement

Heterogeneous Programming and Optimization of Gyrokinetic Toroidal Code and Large-Scale Performance Test on TH-1A.

Nek5000/RS Performance on Advanced GPU Architectures

AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes

Early experiences on the OLCF Frontier system with AthenaPK and Parthenon‐Hydro

Parthenon—a performance portable block-structured adaptive mesh refinement framework

An Extension of the Athena++ Code Framework for Radiation-magnetohydrodynamics in General Relativity Using a Finite-solid-angle Discretization

Phoebus: Performance Portable GRRMHD for Relativistic Astrophysics

Asynchronous-Many-Task Systems: Challenges and Opportunities -- Scaling an AMR Astrophysics Code on Exascale machines using Kokkos and HPX

ExaGRyPE: Numerical General Relativity Solvers Based upon the Hyperbolic PDEs Solver Engine ExaHyPE

GR-Athena++: General-relativistic magnetohydrodynamics simulations of neutron star spacetimes

A Study of Performance Portability in Plasma Physics Simulations

HORIZON: Accelerated General Relativistic Magnetohydrodynamics

Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes

GRaM-X: A new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

A GPU-Accelerated Modern Fortran Version of the ECHO Code for Relativistic Magnetohydrodynamics

Code Optimization on Kepler GPUs and Xeon Phi

Performance Portable Graphics Processing Unit Acceleration of a High-Order Finite Element Multiphysics Application