MatText: Do Language Models Need More than Text & Scale for Materials Modeling?

Nawaf Alampara,Santiago Miret,Kevin Maik Jablonka

2024-06-28

Abstract:Effectively representing materials as text has the potential to leverage the vast advancements of large language models (LLMs) for discovering new materials. While LLMs have shown remarkable success in various domains, their application to materials science remains underexplored. A fundamental challenge is the lack of understanding of how to best utilize text-based representations for materials modeling. This challenge is further compounded by the absence of a comprehensive benchmark to rigorously evaluate the capabilities and limitations of these text representations in capturing the complexity of material systems. To address this gap, we propose MatText, a suite of benchmarking tools and datasets designed to systematically evaluate the performance of language models in modeling materials. MatText encompasses nine distinct text-based representations for material systems, including several novel representations. Each representation incorporates unique inductive biases that capture relevant information and integrate prior physical knowledge about materials. Additionally, MatText provides essential tools for training and benchmarking the performance of language models in the context of materials science. These tools include standardized dataset splits for each representation, probes for evaluating sensitivity to geometric factors, and tools for seamlessly converting crystal structures into text. Using MatText, we conduct an extensive analysis of the capabilities of language models in modeling materials. Our findings reveal that current language models consistently struggle to capture the geometric information crucial for materials modeling across all representations. Instead, these models tend to leverage local information, which is emphasized in some of our novel representations. Our analysis underscores MatText's ability to reveal shortcomings of text-based methods for materials design.

Materials Science,Machine Learning

What problem does this paper attempt to address?

This paper focuses on how to effectively utilize large-scale language models (LLMs) for material modeling in the field of materials science. Currently, although LLMs have shown success in multiple domains, their potential application in materials science has not been fully explored. The main challenges mentioned in the research are the lack of understanding on how to best utilize text-based material representations and the lack of comprehensive benchmarks to evaluate the performance of these representations in capturing the complexity of material systems.

To address these issues, the paper proposes a benchmark tool and dataset called MatText for systematically evaluating the performance of language models on material modeling tasks. MatText includes 9 different text-based representations of material systems, including some novel representations, each incorporating unique inductive biases to capture relevant information and integrate physical knowledge. In addition, MatText provides training and benchmarking tools such as standardized data splits, sensitivity evaluation probes, and tools for converting crystal structures into text.

Through MatText, researchers extensively analyze language models with different representations and data scales, and find that current models generally have difficulties in capturing geometric information and tend to rely more on local information. This indicates limitations of current text-based approaches for material design.

The research also points out that unlike natural language modeling, improving material property prediction may not be simply achieved by expanding parameters or data in existing models like LLMs. The authors believe that the MatText framework will contribute to the design and evaluation of better modeling frameworks.