Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding

Tatsunori Taniai,Ryo Igarashi,Yuta Suzuki,Naoya Chiba,Kotaro Saito,Yoshitaka Ushiku,Kanta Ono

2024-03-18

Abstract:Predicting physical properties of materials from their crystal structures is a fundamental problem in materials science. In peripheral areas such as the prediction of molecular properties, fully connected attention networks have been shown to be successful. However, unlike these finite atom arrangements, crystal structures are infinitely repeating, periodic arrangements of atoms, whose fully connected attention results in infinitely connected attention. In this work, we show that this infinitely connected attention can lead to a computationally tractable formulation, interpreted as neural potential summation, that performs infinite interatomic potential summations in a deeply learned feature space. We then propose a simple yet effective Transformer-based encoder architecture for crystal structures called Crystalformer. Compared to an existing Transformer-based model, the proposed model requires only 29.4% of the number of parameters, with minimal modifications to the original Transformer architecture. Despite the architectural simplicity, the proposed method outperforms state-of-the-art methods for various property regression tasks on the Materials Project and JARVIS-DFT datasets.

Machine Learning,Materials Science,Computational Physics

What problem does this paper attempt to address?

This paper primarily discusses how to effectively utilize the Transformer architecture to address the encoding problem of crystal structures. In materials science, predicting the physical properties from crystal structures is a fundamental challenge. Despite the successful application of fully connected attention networks for predicting molecular properties, the periodicity of crystal structures, with their infinitely repeated atomic arrangements, makes it complex to employ fully connected attention. The paper proposes a novel approach called Crystalformer, which interprets this infinite connection attention as neural potential summation, summing the potential energies between atoms in an abstract feature space. This method simplifies the design of the Transformer encoder for crystal structures, reducing the number of parameters by approximately 29.4% compared to existing Transformer models, while maintaining the simplicity of the architecture. Crystalformer solves the issue of infinite connection by formalizing the attention weights as potential energies that decay with distance, making them computable. Experimental results demonstrate that despite its simplicity, the proposed method outperforms existing state-of-the-art approaches in various attribute regression tasks on the Materials Project and JARVIS-DFT datasets. In summary, the paper attempts to address the periodicity of crystal structures to improve the accuracy of predicting material physical properties. By introducing the Crystalformer model, it achieves more efficient and accurate encoding of crystal structures.

Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding

A periodicity aware transformer for crystal property prediction

Complete and Efficient Graph Transformers for Crystal Material Property Prediction

Study of crystal properties based on attention mechanism and crystal graph convolutional neural network

Neural Structure Fields with Application to Crystal Structure Autoencoders

3D-Transformer: Molecular Representation with Transformer in 3D Space

Improving Crystal Property Prediction from a Multiplex Graph Perspective

CrysToGraph: A Comprehensive Predictive Model for Crystal Materials Properties and the Benchmark

Atomic Positional Embedding-Based Transformer Model for Predicting the Density of States of Crystalline Materials.

Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties

Crystal Transformer Based Universal Atomic Embedding for Accurate and Transferable Prediction of Materials Properties

CrysFormer: Protein Structure Prediction via 3d Patterson Maps and Partial Structure Attention

DeepCrysTet: A Deep Learning Approach Using Tetrahedral Mesh for Predicting Properties of Crystalline Materials

CTGNN: Crystal Transformer Graph Neural Network for Crystal Material Property Prediction

Accelerating material property prediction using generically complete isometry invariants

A Crystal-Specific Pre-Training Framework for Crystal Material Property Prediction

CrysFormer: Protein structure determination via Patterson maps, deep learning, and partial structure attention

Generative Inverse Design of Crystal Structures via Diffusion Models with Transformers

Study of Crystal Property Prediction Based on Dual Attention Mechanism and Transfer Learning

CrysGraphFormer: an equivariant graph transformer for prediction of lattice thermal conductivity with interpretability

Graph theory and graph neural network assisted high-throughput crystal structure prediction and screening for energy conversion and storage