Abstract:Recent advances in Graph Neural Networks (GNNs) have transformed the space of molecular and catalyst discovery. Despite the fact that the underlying physics across these domains remain the same, most prior work has focused on building domain-specific models either in small molecules or in materials. However, building large datasets across all domains is computationally expensive; therefore, the use of transfer learning (TL) to generalize to different domains is a promising but under-explored approach to this problem. To evaluate this hypothesis, we use a model that is pretrained on the Open Catalyst Dataset (OC20), and we study the model's behavior when fine-tuned for a set of different datasets and tasks. This includes MD17, the *CO adsorbate dataset, and OC20 across different tasks. Through extensive TL experiments, we demonstrate that the initial layers of GNNs learn a more basic representation that is consistent across domains, whereas the final layers learn more task-specific features. Moreover, these well-known strategies show significant improvement over the non-pretrained models for in-domain tasks with improvements of 53% and 17% for the *CO dataset and across the Open Catalyst Project (OCP) task, respectively. TL approaches result in up to 4× speedup in model training depending on the target data and task. However, these do not perform well for the MD17 dataset, resulting in worse performance than the non-pretrained model for few molecules. Based on these observations, we propose transfer learning using attentions across atomic systems with graph Neural Networks (TAAG), an attention-based approach that adapts to prioritize and transfer important features from the interaction layers of GNNs. The proposed method outperforms the best TL approach for out-of-domain datasets, such as MD17, and gives a mean improvement of 6% over a model trained from scratch.

Transferable graph neural networks with deep alignment attention

NGAT: Attention in Breadth and Depth Exploration for Semi-Supervised Graph Representation Learning

Deep graph alignment network

T-GAE: Transferable Graph Autoencoder for Network Alignment

Towards Deep Attention in Graph Neural Networks: Problems and Remedies

Transferable graph auto-encoders for cross-network node classification

Transfer learning using attentions across atomic systems with graph neural networks (TAAG)

Graph-Attention-Based Decentralized Edge Learning for Non-IID Data.

GANN: Graph Alignment Neural Network for semi-supervised learning

Global-local graph attention: unifying global and local attention for node classification

Adaptive Transfer Learning on Graph Neural Networks

Neighbor Overlay-Induced Graph Attention Network

Adaptive Graph Diffusion Networks

Efficient Topology-aware Data Augmentation for High-Degree Graph Neural Networks

TANGNN: a Concise, Scalable and Effective Graph Neural Networks with Top-m Attention Mechanism for Graph Representation Learning

Ha-gnn: a novel graph neural network based on hyperbolic attention

A gated graph attention network based on dual graph convolution for node embedding

Can Modifying Data Address Graph Domain Adaptation?

ALEX: Towards Effective Graph Transfer Learning with Noisy Labels

Graph Triple Attention Network: A Decoupled Perspective

NPA: Improving Large-scale Graph Neural Networks with Non-parametric Attention