Abstract:Novel artificial intelligence (AI) technology has expedited various scientific research, e.g., cosmology, physics, and bioinformatics, inevitably becoming a significant category of workload on high-performance computing (HPC) systems. Existing AI benchmarks tend to customize well-recognized AI applications, so as to evaluate the AI performance of HPC systems under the predefined problem size, in terms of datasets and AI models. However, driven by novel AI technology, most of AI applications are evolving fast on models and datasets to achieve higher accuracy and be applicable to more scenarios. Due to the lack of scalability on the problem size, static AI benchmarks might be under competent to help understand the performance trend of evolving AI applications on HPC systems, in particular, the scientific AI applications on large-scale systems. In this paper, we propose a scalable evaluation methodology (SAIH) for analyzing the AI performance trend of HPC systems with scaling the problem sizes of customized AI applications. To enable scalability, SAIH builds a set of novel mechanisms for augmenting problem sizes. As the data and model constantly scale, we can investigate the trend and range of AI performance on HPC systems, and further diagnose system bottlenecks. To verify our methodology, we augment a cosmological AI application to evaluate a real HPC system equipped with GPUs as a case study of SAIH. With data and model augment, SAIH can progressively evaluate the AI performance trend of HPC systems, e.g., increasing from 5.2% to 59.6% of the peak theoretical hardware performance. The evaluation results are analyzed and summarized into insight findings on performance issues. For instance, we find that the AI application constantly consumes the I/O bandwidth of the shared parallel file system during its iteratively training model. If I/O contention exists, the shared parallel file system might become a bottleneck.

Predictive performance and scalability modeling of a large-scale application

Large Scale Numerical Simulation Via Parallelization and Reconfigurable Computing Hardware

Modelling of ASCI High Performance Applications Using PACE

Performance Modelling of Parallel and Distributed Computing Using PACE1

Using Small-Scale History Data to Predict Large-Scale Performance of HPC Application

Performance modeling of parallel and distributed computing using PACE

Experience and Analysis of Scalable High-Fidelity Computational Fluid Dynamics on Modular Supercomputing Architectures

Adaptive Computing for Scale-up Problems

Building a Framework for Predictive Science

Performance Prediction Of Hpc Applications On Intel Processors

SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems

- 1-Application Performance Prediction for the Management and Scheduling of Clusters and Grids

Automated Performance Modeling of HPC Applications Using Machine Learning.

Performance Sensitivity Studies for Strategic Applications

An unstructured CFD mini‐application for the performance prediction of a production CFD code

A directive based hybrid met office NERC cloud model

Adaptive Workload Prediction of Grid Performance in Confidence Windows

Using Hardware Counter-Based Performance Model to Diagnose Scaling Issues of HPC Applications.

Predicting System-level Power for a Hybrid Supercomputer

Parallel Application Performance Prediction Using Analysis Based Models and HPC Simulations

Automatic Parallelization of Sequential Programs