Abstract:The widespread application of multimodal machine learning models like GPT-4 has revolutionized various research fields including computer vision and natural language processing. However, its implementation in materials informatics remains underexplored, despite the presence of materials data across diverse modalities, such as composition and structure. The effectiveness of machine learning models trained on large calculated datasets depends on the accuracy of calculations, while experimental datasets often have limited data availability and incomplete information. This paper introduces a novel approach to multimodal machine learning in materials science via composition-structure bimodal learning. The proposed COmposition-Structure Bimodal Network (COSNet) is designed to enhance learning and predictions of experimentally measured materials properties that have incomplete structure information. Bimodal learning significantly reduces prediction errors across distinct materials properties including Li conductivity in solid electrolyte, band gap, refractive index, dielectric constant, energy, and magnetic moment, surpassing composition-only learning methods. Furthermore, we identified that data augmentation based on modal availability plays a pivotal role in the success of bimodal learning.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper primarily addresses the issue in materials science where experimental datasets often have complete compositional information but incomplete structural information. Specifically, the paper proposes a novel approach to improve the prediction accuracy of experimentally measured material properties by combining compositional and structural bimodal learning. #### Main Contributions 1. **Proposed a Combined Composition-Structure Bimodal Learning Framework**: To improve the learning performance of datasets with incomplete structural information, the researchers proposed the COmposition-Structure Bimodal Network (COSNet) framework. 2. **Demonstrated the Importance of Data Augmentation for Bimodal Learning Improvement**: The study found that data augmentation based on modality availability is crucial for improving bimodal learning, as it ensures that the combined network can fully utilize all available data for training. #### Experimental Results - For four experimentally measured material properties (lithium conductivity, band gap, refractive index, dielectric constant), COSNet reduced the prediction error by approximately 7% to 10% on all complete datasets compared to models using only compositional information, and these improvements were statistically significant. - This improvement was observed not only in data points with structural information but also in those without structural information, indicating that the structural information of some data points can enhance the prediction of other data points lacking structural information. #### Method Overview - The study used four experimental datasets (lithium conductivity, band gap, refractive index, dielectric constant) and two theoretical datasets (total magnetic moment per chemical formula and energy per atom) to test the effectiveness of bimodal learning and COSNet. - In terms of models, ROOST and de-CGCNN were chosen as the compositional and structural networks in COSNet, based on their strong performance on small datasets. - Data augmentation strategies ensured that the combined network could be fully trained, enhancing its predictive performance even for data points without structural information. In summary, the paper aims to improve the prediction of material properties by combining compositional and structural information and demonstrates the effectiveness of this approach on experimental datasets, particularly excelling in cases with incomplete structural information.

Multimodal machine learning for materials science: composition-structure bimodal learning for experimentally measured properties

Group Contribution Method Supervised Neural Network for Precise Design of Organic Nonlinear Optical Materials

Multimodal Learning for Materials

Transfer Learning in Materials Informatics: structure-property relationships through minimal but highly informative multimodal input

UniMat: Unifying Materials Embeddings through Multi-modal Learning

A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials

Advancing materials science through next-generation machine learning

What Information is Necessary and Sufficient to Predict Materials Properties using Machine Learning?

Atomistic graph networks for experimental materials property prediction

Materials Representation and Transfer Learning for Multi-Property Prediction

Material Property Prediction with Element Attribute Knowledge Graphs and Multimodal Representation Learning

MT-CGCNN: Integrating Crystal Graph Convolutional Neural Network with Multitask Learning for Material Property Prediction

Predicting Material Properties Using a 3D Graph Neural Network with Invariant Local Descriptors

Multi-Task Multi-Fidelity Learning of Properties for Energetic Materials

Efficient Surrogate Models for Materials Science Simulations: Machine Learning-based Prediction of Microstructure Properties

Combining Machine Learning Potential and Structure Prediction for Accelerated Materials Design and Discovery

Graph-enhanced deep material network: multiscale materials modeling with microstructural informatics

Learning from 2D: machine learning of 3D effective properties of heterogeneous materials based on 2D microstructure sections

MultiModal-Learning for Predicting Molecular Properties: A Framework Based on Image and Graph Structures

Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning

CrysMMNet: Multimodal Representation for Crystal Property Prediction