Abstract:Knowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities.Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. To guarantee effective knowledge injection, previous studies integrate models with knowledge encoders for representing knowledge retrieved from knowledge graphs. The operations for knowledge retrieval and encoding bring significant computational burdens, restricting the usage of such models in real-world applications that require high inference speed. In this paper, we propose a novel KEPLM named DKPLM that decomposes knowledge injection process of the pre-trained language models in pre-training, fine-tuning and inference stages, which facilitates the applications of KEPLMs in realworld scenarios. Specifically, we first detect knowledge-aware long-tail entities as the target for knowledge injection, enhancing the KEPLMs' semantic understanding abilities and avoiding injecting redundant information. The embeddings of long-tail entities are replaced by "pseudo token representations" formed by relevant knowledge triples. We further design the relational knowledge decoding task for pre-training to force the models to truly understand the injected knowledge by relation triple reconstruction. Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. We further show that DKPLM has a higher inference speed than other competing models due to the decomposing mechanism.

KEML: A Knowledge-Enriched Meta-Learning Framework for Lexical Relation Classification

Knowledge Representation Learning with Entity Descriptions, Hierarchical Types, and Textual Relations

KEHRL: Learning Knowledge-Enhanced Language Representations with Hierarchical Reinforcement Learning

Meta Relational Learning for Few-Shot Link Prediction in Knowledge Graphs

KICE: A Knowledge Consolidation and Expansion Framework for Relation Extraction.

Semi-Supervised Dual Relation Learning for Multi-Label Classification

Meta-Information Guided Meta-Learning for Few-Shot Relation Classification.

Learning from Adjective-Noun Pairs: A Knowledge-enhanced Framework for Target-Oriented Multimodal Sentiment Classification.

TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models

Knowledge-Enhanced Prototypical Network with Structural Semantics for Few-Shot Relation Classification.

MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation

Improving Relation Extraction by Knowledge Representation Learning

KNET: A General Framework for Learning Word Embedding using Morphological Knowledge

Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension

Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition

Language Models as Knowledge Embeddings

KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs

Attention-Based Lstm With Filter Mechanism For Entity Relation Classification

Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding

Meta-Learning Based Dynamic Adaptive Relation Learning for Few-Shot Knowledge Graph Completion

DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding