Abstract:Large language models (LLMs) have recently transformed both the academic and industrial landscapes due to their remarkable capacity to understand, analyze, and generate texts based on their vast knowledge and reasoning ability. Nevertheless, one major drawback of LLMs is their substantial computational cost for pre-training due to their unprecedented amounts of parameters. The disadvantage is exacerbated when new knowledge frequently needs to be introduced into the pre-trained model. Therefore, it is imperative to develop effective and efficient techniques to update pre-trained LLMs. Traditional methods encode new knowledge in pre-trained LLMs through direct fine-tuning. However, naively re-training LLMs can be computationally intensive and risks degenerating valuable pre-trained knowledge irrelevant to the update in the model. Recently, Knowledge-based Model Editing (KME) has attracted increasing attention, which aims to precisely modify the LLMs to incorporate specific knowledge, without negatively influencing other irrelevant knowledge. In this survey, we aim to provide a comprehensive and in-depth overview of recent advances in the field of KME. We first introduce a general formulation of KME to encompass different KME strategies. Afterward, we provide an innovative taxonomy of KME techniques based on how the new knowledge is introduced into pre-trained LLMs, and investigate existing KME strategies while analyzing key insights, advantages, and limitations of methods from each category. Moreover, representative metrics, datasets, and applications of KME are introduced accordingly. Finally, we provide an in-depth analysis regarding the practicality and remaining challenges of KME and suggest promising research directions for further advancement in this field.

A Survey of Knowledge Enhanced Pre-trained Models

A Survey of Knowledge Enhanced Pre-trained Language Models

A Survey on Knowledge-Enhanced Pre-trained Language Models

Pre-trained models for natural language processing: A survey

Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining

Structure Pre-training and Prompt Tuning for Knowledge Graph Transfer

DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Knowledge Extraction in Low-Resource Scenarios: Survey and Perspective

The Life Cycle of Knowledge in Big Language Models: A Survey

Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach.

TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models

Pretrain-KGE - Learning Knowledge Representation from Pretrained Language Models.

Pre-Trained Language Models for Keyphrase Prediction: A Review

Knowledge Editing for Large Language Models: A Survey

Pre-Trained Models: Past, Present and Future

Pre-Trained Language Models and Their Applications

An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering