Fast and Continual Knowledge Graph Embedding via Incremental LoRA

Jiajun Liu,Wenjun Ke,Peng Wang,Jiahao Wang,Jinhua Gao,Ziyu Shang,Guozheng Li,Zijie Xu,Ke Ji,Yining Li

2024-07-08

Abstract:Continual Knowledge Graph Embedding (CKGE) aims to efficiently learn new knowledge and simultaneously preserve old knowledge. Dominant approaches primarily focus on alleviating catastrophic forgetting of old knowledge but neglect efficient learning for the emergence of new knowledge. However, in real-world scenarios, knowledge graphs (KGs) are continuously growing, which brings a significant challenge to fine-tuning KGE models efficiently. To address this issue, we propose a fast CKGE framework (\model), incorporating an incremental low-rank adapter (\mec) mechanism to efficiently acquire new knowledge while preserving old knowledge. Specifically, to mitigate catastrophic forgetting, \model\ isolates and allocates new knowledge to specific layers based on the fine-grained influence between old and new KGs. Subsequently, to accelerate fine-tuning, \model\ devises an efficient \mec\ mechanism, which embeds the specific layers into incremental low-rank adapters with fewer training parameters. Moreover, \mec\ introduces adaptive rank allocation, which makes the LoRA aware of the importance of entities and adjusts its rank scale adaptively. We conduct experiments on four public datasets and two new datasets with a larger initial scale. Experimental results demonstrate that \model\ can reduce training time by 34\%-49\% while still achieving competitive link prediction performance against state-of-the-art models on four public datasets (average MRR score of 21.0\% vs. 21.1\%).Meanwhile, on two newly constructed datasets, \model\ saves 51\%-68\% training time and improves link prediction performance by 1.5\%.

Artificial Intelligence

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper aims to address the issue of efficiently updating and maintaining old knowledge in Knowledge Graph Embedding (KGE) as knowledge graphs continue to grow. Specifically, the paper proposes a fast continuous knowledge graph embedding framework (FastKGE) that efficiently acquires new knowledge while preserving old knowledge through an incremental low-rank adapter (IncLoRA) mechanism.

#### Main Contributions:
1. **Introduction of Low-Rank Adapter**: For the first time, low-rank adapters (LoRA) are introduced into the continuous knowledge graph embedding task, allowing new knowledge to be stored in low-rank adapters, thereby reducing training costs and effectively preserving old knowledge.
2. **Proposing the FastKGE Framework**: This framework mitigates catastrophic forgetting by isolating new knowledge to specific layers and reduces training costs using the incremental low-rank adapter (IncLoRA) mechanism.
3. **Experimental Validation**: Experiments show that FastKGE significantly reduces training time and performs comparably to state-of-the-art models in link prediction tasks. Additionally, two new large-scale initial knowledge graph datasets (FB-CKGE and WN-CKGE) are released.