Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction

Qi Sun,Kun Huang,Xiaocui Yang,Rong Tong,Kun Zhang,Soujanya Poria

2024-01-25

Abstract:Document-level Relation Triplet Extraction (DocRTE) is a fundamental task in information systems that aims to simultaneously extract entities with semantic relations from a document. Existing methods heavily rely on a substantial amount of fully labeled data. However, collecting and annotating data for newly emerging relations is time-consuming and labor-intensive. Recent advanced Large Language Models (LLMs), such as ChatGPT and LLaMA, exhibit impressive long-text generation capabilities, inspiring us to explore an alternative approach for obtaining auto-labeled documents with new relations. In this paper, we propose a Zero-shot Document-level Relation Triplet Extraction (ZeroDocRTE) framework, which generates labeled data by retrieval and denoising knowledge from LLMs, called GenRDK. Specifically, we propose a chain-of-retrieval prompt to guide ChatGPT to generate labeled long-text data step by step. To improve the quality of synthetic data, we propose a denoising strategy based on the consistency of cross-document knowledge. Leveraging our denoised synthetic data, we proceed to fine-tune the LLaMA2-13B-Chat for extracting document-level relation triplets. We perform experiments for both zero-shot document-level relation and triplet extraction on two public datasets. The experimental results illustrate that our GenRDK framework outperforms strong baselines.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the zero-shot learning problem in the task of document-level relation triplet extraction (DocRTE). Specifically, the authors propose a new framework called GenRDK, which generates automatically annotated documents by retrieving and denoising implicit knowledge from large language models (such as ChatGPT and LLaMA). This approach is particularly suitable for handling emerging types of relations, as collecting and annotating data for these new relations is both time-consuming and labor-intensive.

The main contributions of the paper are as follows:
1. Proposing a new framework for the zero-shot document-level relation triplet extraction (ZeroDocRTE) task, which can generate synthetic data by retrieving and denoising implicit knowledge from large language models.
2. Designing a chain-of-retrieval prompt to guide ChatGPT in gradually generating documents that contain complex semantic contexts and multiple relation triplets.
3. Proposing a cross-document knowledge denoising strategy based on consistency to improve the quality of synthetic data, reduce unreliable relational facts, and supplement missing relational facts.
4. Conducting zero-shot document-level relation and triplet extraction experiments on 2 public datasets, showing that the GenRDK framework significantly outperforms baseline models in terms of performance.

Through these methods, the researchers hope to effectively tackle the task of document-level relation triplet extraction for new relation types without the need for extensive manual annotation.