Abstract:Effectively representing materials as text has the potential to leverage the vast advancements of large language models (LLMs) for discovering new materials. While LLMs have shown remarkable success in various domains, their application to materials science remains underexplored. A fundamental challenge is the lack of understanding of how to best utilize text-based representations for materials modeling. This challenge is further compounded by the absence of a comprehensive benchmark to rigorously evaluate the capabilities and limitations of these text representations in capturing the complexity of material systems. To address this gap, we propose MatText, a suite of benchmarking tools and datasets designed to systematically evaluate the performance of language models in modeling materials. MatText encompasses nine distinct text-based representations for material systems, including several novel representations. Each representation incorporates unique inductive biases that capture relevant information and integrate prior physical knowledge about materials. Additionally, MatText provides essential tools for training and benchmarking the performance of language models in the context of materials science. These tools include standardized dataset splits for each representation, probes for evaluating sensitivity to geometric factors, and tools for seamlessly converting crystal structures into text. Using MatText, we conduct an extensive analysis of the capabilities of language models in modeling materials. Our findings reveal that current language models consistently struggle to capture the geometric information crucial for materials modeling across all representations. Instead, these models tend to leverage local information, which is emphasized in some of our novel representations. Our analysis underscores MatText's ability to reveal shortcomings of text-based methods for materials design.

Evaluating Large Language Models for Material Selection

Evaluating Large Language Models for Material Selection

Beyond designer's knowledge: Generating materials design hypotheses via large language models

A Prompt-Engineered Large Language Model, Deep Learning Workflow for Materials Classification

Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

LLMatDesign: Autonomous Materials Discovery with Large Language Models

Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions

Materials science in the era of large language models: a perspective

DARWIN 1.5: Large Language Models as Materials Science Adapted Learners

MSEval: A Dataset for Material Selection in Conceptual Design to Evaluate Algorithmic Models

From Text to Insight: Large Language Models for Materials Science Data Extraction

Are LLMs Ready for Real-World Materials Discovery?

How Can Large Language Models Help Humans in Design and Manufacturing?

Interpretable Machine Learning for Materials Design

Mining experimental data from Materials Science literature with Large Language Models: an evaluation study

Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models

Generative large language models in engineering design: opportunities and challenges

MatText: Do Language Models Need More than Text & Scale for Materials Modeling?

Machine learning-assisted design of material properties