Efficient Tuning and Inference for Large Language Models on Textual Graphs

Yun Zhu,Yaoke Wang,Haizhou Shi,Siliang Tang

2024-07-24

Abstract:Rich textual and topological information of textual graphs need to be modeled in real-world applications such as webpages, e-commerce, and academic articles. Practitioners have been long following the path of adopting a shallow text encoder and a subsequent graph neural network (GNN) to solve this problem. In light of recent advancements in large language models (LLMs), it is apparent that integrating LLMs for enhanced textual encoding can substantially improve the performance of textual graphs. Nevertheless, the efficiency of these methods poses a significant challenge. In this paper, we propose ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs with an LLM encoder. The key insight is to combine the LLMs and GNNs through a tunable side structure, which significantly reduces the training complexity without impairing the joint model's capacity. Extensive experiments on textual graphs demonstrate our method's effectiveness by achieving the best model performance, meanwhile having the lowest training cost compared to previous methods. Moreover, we introduce two variants with caching and dynamic early exit to further enhance training and inference speed. Specifically, caching accelerates ENGINE's training by 12x, and dynamic early exit achieves up to 5x faster inference with a negligible performance drop (at maximum 1.17% relevant drop across 7 datasets). Our codes are available at: <a class="link-external link-https" href="https://github.com/ZhuYun97/ENGINE" rel="external noopener nofollow">this https URL</a>

Computation and Language

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily addresses the issue of efficient fine-tuning and inference using large language models (LLMs) on textual graphs. Specifically:

1. **Shortcomings of existing methods**:
- Current methods typically combine shallow text encoders with graph neural networks (GNNs) to handle textual graph data. However, these methods struggle to capture complex semantic relationships, leading to limited performance.
- Although combining LLMs with GNNs can significantly improve performance, this joint training incurs substantial memory and time costs.

2. **Proposed new method**:
- The paper introduces a method called ENGINE, a parameter and memory-efficient fine-tuning algorithm for integrating LLMs on textual graphs.
- ENGINE introduces a tunable side structure, G-Ladder, which significantly reduces training complexity without compromising the overall model capability.
- Additionally, the paper presents two variants: one that uses caching to accelerate the training process, and another that employs a dynamic early exit mechanism during inference to further enhance efficiency.

3. **Experimental results**:
- Experiments show that ENGINE outperforms existing methods across multiple datasets while maintaining the lowest training costs.
- The variant using caching speeds up training by 12 times, and the variant with the dynamic early exit mechanism speeds up inference by up to 5 times, with minimal performance loss (maximum of 1.17%).

In summary, this paper aims to address key challenges in textual graph analysis by efficiently integrating LLMs and GNNs, thereby improving model performance and reducing resource consumption.