Abstract:The Parallel Ocean Program (POP), the ocean model component of the Community Earth System Model (CESM), is widely used in climate research. Most current work in CESM-POP focuses on improving the model's efficiency or accuracy, such as improving numerical methods, advancing parameterization, porting to new architectures, or increasing parallelism. Since ocean dynamics are chaotic in nature, achieving bit-for-bit (BFB) identical results in ocean solutions cannot be guaranteed for even tiny code modifications, and determining whether modifications are admissible (i.e., statistically consistent with the original results) is non-trivial. In recent work, an ensemble-based statistical approach was shown to work well for software verification (i.e., quality assurance) on atmospheric model data. The general idea of the ensemble-based statistical consistency testing is to use a qualitative measurement of the variability of the ensemble of simulations as a metric with which to compare future simulations and make a determination of statistical distinguishability. The capability to determine consistency without BFB results boosts model confidence and provides the flexibility needed, for example, for more aggressive code optimizations and the use of heterogeneous execution environments. Since ocean and atmosphere models have differing characteristics in term of dynamics, spatial variability, and timescales, we present a new statistical method to evaluate ocean model simulation data that requires the evaluation of ensemble means and deviations in a spatial manner. In particular, the statistical distribution from an ensemble of CESM-POP simulations is used to determine the standard score of any new model solution at each grid point. Then the percentage of points that have scores greater than a specified threshold indicates whether the new model simulation is statistically distinguishable from the ensemble simulations. Both ensemble size and composition are important. Our experiments indicate that the new POP ensemble consistency test (POP-ECT) tool is capable of distinguishing cases that should be statistically consistent with the ensemble and those that should not, as well as providing a simple, subjective and systematic way to detect errors in CESM-POP due to the hardware or software stack, positively contributing to quality assurance for the CESM-POP code.

HECT: High-Dimensional Ensemble Consistency Testing for Climate Models

Achieving Robust and Transferable Performance for Conservation‐Based Models of Dynamical Physical Systems

A Deep Learning-Based Consistency Test Approach for Earth System Models on Heterogeneous Many-Core Systems

The Ensemble Consistency Test: From CESM to MPAS and Beyond

Evaluating Statistical Consistency in the Ocean Model Component of the Community Earth System Model (pycect V2.0)

A non-intrusive machine learning framework for debiasing long-time coarse resolution climate simulations and quantifying rare events statistics

Exploiting large ensembles for a better yet simpler climate model evaluation

ACE: A fast, skillful learned global atmospheric model for climate prediction

A Non‐Intrusive Machine Learning Framework for Debiasing Long‐Time Coarse Resolution Climate Simulations and Quantifying Rare Events Statistics

A Forecast Test for Reducing Dynamical Dimensionality of Model Emulators

Modes of Variability in E3SM and CESM Large Ensembles

Probabilistic Emulation of a Global Climate Model with Spherical DYffusion

A Deep Learning Earth System Model for Stable and Efficient Simulation of the Current Climate

Stress-testing the coupled behavior of hybrid physics-machine learning climate simulations on an unseen, warmer climate

DiffESM: Conditional Emulation of Temperature and Precipitation in Earth System Models with 3D Diffusion Models

Short Ensembles: an Efficient Method for Discerning Climate-Relevant Sensitivities in Atmospheric General Circulation Models

Climate-Invariant Machine Learning

Metamorphic testing of machine learning and conceptual hydrologic models

PACER: Physics Informed Uncertainty Aware Climate Emulator

Scalable Sensitivity and Uncertainty Analysis for Causal-Effect Estimates of Continuous-Valued Interventions