All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks

Ajay Jaiswal,Nurendra Choudhary,Ravinarayana Adkathimar,Muthu P. Alagappan,Gaurush Hiranandani,Ying Ding,Zhangyang Wang,Edward W Huang,Karthik Subbian

2024-07-21

Abstract:Graph Neural Networks (GNNs) have attracted immense attention in the past decade due to their numerous real-world applications built around graph-structured data. On the other hand, Large Language Models (LLMs) with extensive pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data. In this paper, we investigate how LLMs can be leveraged in a computationally efficient fashion to benefit rich graph-structured data, a modality relatively unexplored in LLM literature. Prior works in this area exploit LLMs to augment every node features in an ad-hoc fashion (not scalable for large graphs), use natural language to describe the complex structural information of graphs, or perform computationally expensive finetuning of LLMs in conjunction with GNNs. We propose E-LLaGNN (Efficient LLMs augmented GNNs), a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph. More specifically, E-LLaGNN relies on sampling high-quality neighborhoods using LLMs, followed by on-demand neighborhood feature enhancement using diverse prompts from our prompt catalog, and finally information aggregation using message passing from conventional GNN architectures. We explore several heuristics-based active node selection strategies to limit the computational and memory footprint of LLMs when handling millions of nodes. Through extensive experiments & ablation on popular graph benchmarks of varying scales (Cora, PubMed, ArXiv, & Products), we illustrate the effectiveness of our E-LLaGNN framework and reveal many interesting capabilities such as improved gradient flow in deep GNNs, LLM-free inference ability etc.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively utilize large - language models (LLMs) to enhance the message - passing process of graph neural networks (GNNs) in the case of limited computing resources, so as to improve the ability of GNNs to process large - scale graph - structured data. Specifically, the paper proposes a framework named E - LLaGNN. This framework selects high - quality neighborhoods by on - demand invocation of LLMs services and enhances node text information through a diverse prompt catalogue, thereby embedding rich world knowledge during the training process. In addition, E - LLaGNN also explores a variety of heuristic - based active node selection strategies to limit the computing and memory footprint of LLMs when processing millions of nodes, and finally achieves inference capabilities without relying on LLMs while maintaining performance.

The main contributions of the paper include:
1. Proposing the E - LLaGNN framework, which can dynamically integrate the capabilities of LLMs into GNNs according to the computing budget during training and support the inference process without LLMs participation.
2. Providing a variety of active node selection metrics for on - demand enhancement during E - LLaGNN training. Experiments show how these metrics translate into performance improvements, making it possible to integrate LLMs with industrial - scale graph data.
3. Extensive experiments and ablation studies show that E - LLaGNN can achieve superior performance in popular graph benchmarks, revealing useful insights such as that LLM enhancement can improve the gradient flow in deep GNN backbones.