Abstract:Pre-trained transformer language models on large unlabeled corpus have produced state-of-the-art results in natural language processing, organic molecule design, and protein sequence generation. However, no such models have been applied to learn the composition patterns of inorganic materials. Here we train a series of seven modern transformer language models (GPT, GPT-2, GPT-Neo, GPT-J, BLMM, BART, and RoBERTa) using the expanded formulas from material deposited in the ICSD, OQMD, and Materials Projects databases. Six different datasets with/out non-charge-neutral or balanced electronegativity samples are used to benchmark the performances and uncover the generation biases of modern transformer models for the generative design of materials compositions. Our extensive experiments showed that the causal language models based materials transformers can generate chemically valid materials compositions with as high as 97.54\% to be charge neutral and 91.40\% to be electronegativity balanced, which has more than 6 times higher enrichment compared to a baseline pseudo-random sampling algorithm. These models also demonstrate high novelty and their potential in new materials discovery has been proved by their capability to recover the leave-out materials. We also find that the properties of the generated samples can be tailored by training the models with selected training sets such as high-bandgap materials. Our experiments also showed that different models each have their own preference in terms of the properties of the generated samples and their running time complexity varies a lot. We have applied our materials transformer models to discover a set of new materials as validated using DFT calculations.

Generative adversarial networks (GAN) based efficient sampling of chemical composition space for inverse design of inorganic materials

Generative adversarial networks (GAN) based efficient sampling of chemical space for inverse design of inorganic materials

A Generative Adversarial Networks (GAN) based efficient sampling method for inverse design of metallic glasses

Active learning based generative design for the discovery of wide bandgap materials

Generative Models for Automatic Chemical Design

Inverse design of two-dimensional graphene/h-BN hybrids by a regressional and conditional GAN

High‐Throughput Discovery of Novel Cubic Crystal Materials Using Deep Generative Neural Networks

Inverse Design of Semiconductor Materials with Deep Generative Models

NSGAN: a non-dominant sorting optimisation-based generative adversarial design framework for alloy discovery

Generative Design of inorganic compounds using deep diffusion language models

Generative deep learning for the inverse design of materials

Inverse Structural Design of Graphene/Boron Nitride Hybrids by Regressional GAN

Deep Generative Models for Materials Discovery and Machine Learning-Accelerated Innovation

MatterGen: a generative model for inorganic materials design

Applications of Generative Adversarial Networks in Materials Science

Materials Transformers Language Models for Generative Materials Design: a benchmark study

MatterGPT: A Generative Transformer for Multi-Property Inverse Design of Solid-State Materials

Inverse Design with Deep Generative Models: Next Step in Materials Discovery

Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attention

Enhancing molecular design efficiency: Uniting language models and generative networks with genetic algorithms