Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs

Yijian Qin,Xin Wang,Ziwei Zhang,Wenwu Zhu

2024-03-10

Abstract:Text-attributed graphs (TAGs) are prevalent on the web and research over TAGs such as citation networks, e-commerce networks and social networks has attracted considerable attention in the web community. Recently, large language models (LLMs) have demonstrated exceptional capabilities across a wide range of tasks. However, the existing works focus on harnessing the potential of LLMs solely relying on prompts to convey graph structure information to LLMs, thus suffering from insufficient understanding of the complex structural relationships within TAGs. To address this problem, in this paper we present the Disentangled Graph-Text Learner (DGTL) model, which is able to enhance the reasoning and predicting capabilities of LLMs for TAGs. Our proposed DGTL model incorporates graph structure information through tailored disentangled graph neural network (GNN) layers, enabling LLMs to capture the intricate relationships hidden in text-attributed graphs from multiple structural factors. Furthermore, DGTL operates with frozen pre-trained LLMs, reducing computational costs and allowing much more flexibility in combining with different LLM models. Experimental evaluations demonstrate the effectiveness of the proposed DGTL model on achieving superior or comparable performance over state-of-the-art baselines. Additionally, we also demonstrate that our DGTL model can offer natural language explanations for predictions, thereby significantly enhancing model interpretability.

Computation and Language,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to more effectively use large - language models (LLMs) in text - attributed graphs (TAGs) to enhance the understanding and prediction ability of complex structural relationships. Specifically, existing methods mainly rely on prompts to convey graph - structure information to LLMs, which leads to an insufficient understanding of the complex structural relationships within TAGs. To solve this problem, the paper proposes a model named Disentangled Graph - Text Learner (DGTL), aiming to integrate graph - structure information through customized disentangled graph neural network layers, so that LLMs can capture the complex relationships in text - attributed graphs and extract information from multiple structural factors. In addition, the DGTL model allows pre - trained LLMs to remain frozen, thereby reducing computational costs and increasing the flexibility of combination with different LLM models. Experimental evaluations show that the proposed DGTL model is effective in achieving performance superior to or comparable to that of the existing state - of - the - art baseline models. At the same time, the DGTL model can also provide natural - language explanations, significantly enhancing the interpretability of the model.