Abstract:During early optimization passes, compilers must make predictions for machine-dependent characteristics such as execution unit utilization, number of register spills, latency, throughput etc. to generate better code. Often a hand-written static/analytical hardware cost model is built into the compiler. However, the need for more sophisticated and varied predictions has become more pronounced with the development of deep learning compilers which need to optimize dataflow graphs. Such compilers usually employ a much higher level MLIR form as an IR representation before lowering to traditional LLVM-IR. A static/analytical cost model in such a scenario is cumbersome and error prone as the opcodes represent very high level algebraic/arithmetic operations. Hence, we develop a machine learning-based cost model for high-level MLIR which can predict different target variables of interest such as CPU/GPU/xPU utilization, instructions executed, register usage etc. By considering the incoming MLIR as a text input a la NLP models we can apply well-known techniques from modern NLP research to help predict hardware characteristics more accurately. We expect such precise ML-driven hardware cost models to guide our deep learning compiler in graph level optimizations around operator fusion, local memory allocation, kernel scheduling etc. as well as in many kernel-level optimizations such as loop interchange, LICM and unroll. We report early work-in -progress results of developing such models on high-level MLIR representing dataflow graphs emitted by Pytorch/Tensorflow-like frameworks as well as lower-level dialects like affine. We show that these models can provide reasonably good estimates with low error bounds for various hardware characteristics of interest and can be a go-to mechanism for hardware cost modelling in the future.

HPAC-ML: A Programming Model for Embedding ML Surrogates in Scientific Applications

Scientific Machine Learning Enables Multiphysics Digital Twins of Large-Scale Electronic Chips

PySCIPOpt-ML: Embedding Trained Machine Learning Models into Mixed-Integer Programs

Integrating Machine Learning with HPC-driven Simulations for Enhanced Student Learning

A New Certified Hierarchical and Adaptive RB-ML-ROM Surrogate Model for Parametrized PDEs

Machine learning surrogates for efficient hydrologic modeling: Insights from stochastic simulations of managed aquifer recharge

Demystifying the Data Need of ML-surrogates for CFD Simulations

Using machine learning as a surrogate model for agent-based simulations

Applications of ML-Based Surrogates in Bayesian Approaches to Inverse Problems

Understanding ML driven HPC: Applications and Infrastructure

Machine Learning Surrogate Modeling for Meshless Methods: Leveraging Universal Approximation

Programming with Neural Surrogates of Programs

Advances in surrogate based modeling, feasibility analysis, and optimization: A review

HPC-Coder: Modeling Parallel Programs using Large Language Models

A Single-Fidelity Surrogate Modeling Method Based on Nonlinearity Integrated Multi-Fidelity Surrogate

Efficient Learning of Accurate Surrogates for Simulations of Complex Systems

LM4HPC: Towards Effective Language Model Application in High-Performance Computing

A Fortran-Python Interface for Integrating Machine Learning Parameterization into Earth System Models

ML-driven Hardware Cost Model for MLIR

Promising and worth-to-try future directions for advancing state-of-the-art surrogates methods of agent-based models in social and health computational sciences

MLModelScope: A Distributed Platform for Model Evaluation and Benchmarking at Scale