Abstract:Knowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities.Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. To guarantee effective knowledge injection, previous studies integrate models with knowledge encoders for representing knowledge retrieved from knowledge graphs. The operations for knowledge retrieval and encoding bring significant computational burdens, restricting the usage of such models in real-world applications that require high inference speed. In this paper, we propose a novel KEPLM named DKPLM that decomposes knowledge injection process of the pre-trained language models in pre-training, fine-tuning and inference stages, which facilitates the applications of KEPLMs in realworld scenarios. Specifically, we first detect knowledge-aware long-tail entities as the target for knowledge injection, enhancing the KEPLMs' semantic understanding abilities and avoiding injecting redundant information. The embeddings of long-tail entities are replaced by "pseudo token representations" formed by relevant knowledge triples. We further design the relational knowledge decoding task for pre-training to force the models to truly understand the injected knowledge by relation triple reconstruction. Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. We further show that DKPLM has a higher inference speed than other competing models due to the decomposing mechanism.

Decouple Knowledge from Paramters for Plug-and-play Language Modeling.

Decouple knowledge from parameters for plug-and-play language modeling

Plug-and-Play Knowledge Injection for Pre-trained Language Models

DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

MiniPLM: Knowledge Distillation for Pre-Training Language Models

Knowledge Inheritance for Pre-trained Language Models

From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models

DIEKAE: Difference Injection for Efficient Knowledge Augmentation and Editing of Large Language Models

Self-Updatable Large Language Models with Parameter Integration

Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding

Knowledge Rumination for Pre-trained Language Models

Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models

K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce

Augmented Large Language Models with Parametric Knowledge Guiding

A Survey on Knowledge-Enhanced Pre-trained Language Models

Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective

Scaling Pre-trained Language Models to Deeper Via Parameter-efficient Architecture

TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models

Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning