Abstract:We present a benchmark test suite and an automated machine learning procedure for evaluating supervised machine learning (ML) models for predicting properties of inorganic bulk materials. The test suite, Matbench, is a set of 13 ML tasks that range in size from 312 to 132k samples and contain data from 10 density functional theory-derived and experimental sources. Tasks include predicting optical, thermal, electronic, thermodynamic, tensile, and elastic properties given a materials composition and/or crystal structure. The reference algorithm, Automatminer, is a highly-extensible, fully-automated ML pipeline for predicting materials properties from materials primitives (such as composition and crystal structure) without user intervention or hyperparameter tuning. We test Automatminer on the Matbench test suite and compare its predictive power with state-of-the-art crystal graph neural networks and a traditional descriptor-based Random Forest model. We find Automatminer achieves the best performance on 8 of 13 tasks in the benchmark. We also show our test suite is capable of exposing predictive advantages of each algorithm - namely, that crystal graph methods appear to outperform traditional machine learning methods given ~10^4 or greater data points. The pre-processed, ready-to-use Matbench tasks and the Automatminer source code are open source and available online (<a class="link-external link-http" href="http://hackingmaterials.lbl.gov/automatminer/" rel="external noopener nofollow">this http URL</a>). We encourage evaluating new materials ML algorithms on the MatBench benchmark and comparing them against the latest version of Automatminer.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the lack of standards for evaluating and comparing machine - learning models in the field of materials science. Specifically: 1. **Lack of systematic methods to compare and select machine - learning models in materials science**: In the field of materials science, although there are some commonly used materials data sets, different algorithms are usually not tested on the same data set, and data pre - processing methods also vary. This makes it difficult to compare models and hinders the rapid development of this field. 2. **Need for a standardized benchmark test suite**: In order to promote the development of machine - learning models in materials science, a benchmark test suite that includes multiple material property prediction tasks is required, so that researchers can systematically evaluate and compare different machine - learning models. 3. **Need for an automated reference algorithm**: Due to the wide variety of machine - learning tasks in materials science, manually designing and tuning models is both time - consuming and error - prone. Therefore, an automated and general - purpose machine - learning pipeline is required, which can automatically generate high - quality prediction models according to a given data set without expert intervention. To solve the above problems, the author makes two main contributions: - **Matbench test suite**: This is a collection of 13 data - mining tasks specific to materials science, covering data sets of different scales from a few hundred samples to more than 100,000 samples. These tasks involve the prediction of various material properties such as optics, thermology, electronics, thermodynamics, tensile and elasticity. - **Automatminer reference algorithm**: This is a highly scalable and fully automated machine - learning pipeline that can automatically generate models for predicting material properties from the basic properties of materials (such as composition and crystal structure). Automatminer achieves efficient prediction of material properties through steps such as generating descriptors, feature dimension reduction, data pre - processing and model selection. Through these two tools, the author hopes to provide a standard benchmark test platform for machine - learning research in the materials science field, thereby accelerating innovation and development in this field.

Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm

Matminer: an Open Source Toolkit for Materials Data Mining

LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction

MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling

Matbench Discovery -- A framework to evaluate machine learning crystal stability predictions

Leveraging Large-scale Computational Database and Deep Learning for Accurate Prediction of Material Properties

Auto-MatRegressor: liberating machine learning alchemists

Machine Learning Materials Properties with Accurate Predictions, Uncertainty Estimates, Domain Guidance, and Persistent Online Accessibility

A critical examination of robustness and generalizability of machine learning prediction of materials properties

Improving Material Property Prediction by Leveraging the Large-Scale Computational Database and Deep Learning

Benchmarking Active Learning Strategies for Materials Optimization and Discovery

JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods

Benchmarking AutoML for regression tasks on small tabular data in materials design

Structure-based out-of-distribution (OOD) materials property prediction: a benchmark study

Harnessing the Materials Project for machine-learning and accelerated discovery

Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models

Assessing data-driven predictions of band gap and electrical conductivity for transparent conducting materials

MechProNet: Machine Learning Prediction of Mechanical Properties in Metal Additive Manufacturing

LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions

NJmat: Data-Driven Machine Learning Interface to Accelerate Material Design

NOMAD 2018 Kaggle Competition: Solving Materials Science Challenges Through Crowd Sourcing