KEHRL: Learning Knowledge-Enhanced Language Representations with Hierarchical Reinforcement Learning

Dongyang Li,Taolin Zhang,Longtao Huang,Chengyu Wang,Xiaofeng He,Hui Xue

2024-06-24

Abstract:Knowledge-enhanced pre-trained language models (KEPLMs) leverage relation triples from knowledge graphs (KGs) and integrate these external data sources into language models via self-supervised learning. Previous works treat knowledge enhancement as two independent operations, i.e., knowledge injection and knowledge integration. In this paper, we propose to learn Knowledge-Enhanced language representations with Hierarchical Reinforcement Learning (KEHRL), which jointly addresses the problems of detecting positions for knowledge injection and integrating external knowledge into the model in order to avoid injecting inaccurate or irrelevant knowledge. Specifically, a high-level reinforcement learning (RL) agent utilizes both internal and prior knowledge to iteratively detect essential positions in texts for knowledge injection, which filters out less meaningful entities to avoid diverting the knowledge learning direction. Once the entity positions are selected, a relevant triple filtration module is triggered to perform low-level RL to dynamically refine the triples associated with polysemic entities through binary-valued actions. Experiments validate KEHRL's effectiveness in probing factual knowledge and enhancing the model's performance on various natural language understanding tasks.

Computation and Language

What problem does this paper attempt to address?

The paper attempts to address two main challenges faced when effectively injecting external knowledge graph information into pre-trained language models (PLMs) to enhance semantic representation capabilities:

1. **Selection of Knowledge Injection Positions**: How to choose appropriate positions in the text for knowledge injection, avoiding the injection of redundant or irrelevant knowledge into entities that have already been sufficiently learned.
2. **Integration of External Knowledge**: How to effectively filter out accurate relational triples related to polysemous words from the knowledge graph and avoid introducing inaccurate or irrelevant knowledge.

The paper proposes a hierarchical reinforcement learning-based method (KEHRL) to solve the above problems by dynamically detecting entity injection positions and filtering relevant relational triples, thereby improving the model's performance in various natural language understanding tasks.