ChatUIE: Exploring Chat-based Unified Information Extraction using Large Language Models

Jun Xu,Mengshu Sun,Zhiqiang Zhang,Jun Zhou

2024-03-08

Abstract:Recent advancements in large language models have shown impressive performance in general chat. However, their domain-specific capabilities, particularly in information extraction, have certain limitations. Extracting structured information from natural language that deviates from known schemas or instructions has proven challenging for previous prompt-based methods. This motivated us to explore domain-specific modeling in chat-based language models as a solution for extracting structured information from natural language. In this paper, we present ChatUIE, an innovative unified information extraction framework built upon ChatGLM. Simultaneously, reinforcement learning is employed to improve and align various tasks that involve confusing and limited samples. Furthermore, we integrate generation constraints to address the issue of generating elements that are not present in the input. Our experimental results demonstrate that ChatUIE can significantly improve the performance of information extraction with a slight decrease in chatting ability.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Existing large - language models perform well in general - purpose chatting, but have certain limitations in information extraction tasks in specific fields, especially when it is necessary to extract structured information from natural language and this information deviates from known patterns or instructions. This is mainly because previous prompt - based methods perform poorly in handling zero - sample tasks, and especially when encountering unseen patterns, their performance will decline significantly. In addition, these methods often rely on predefined patterns or fixed instructions, which limits their ability to support natural - language extraction.

To overcome these problems, the paper proposes **ChatUIE**, an innovative unified information extraction framework based on ChatGLM. This framework improves information extraction tasks through the following aspects:

1. **Domain - Specific Modeling**: Incorporate domain knowledge into the large - language model by using Supervised Fine - Tuning (SFT).
2. **Reinforcement Learning**: Introduce Reinforcement Learning (RL) to improve and align various tasks, especially those that include confusion and limited samples.
3. **Generation Constraint**: Adopt Generation Constraint Decoding to ensure that the generated content is within the input range and avoid generating elements not in the input.

Through these methods, ChatUIE can significantly improve the performance of information extraction tasks while maintaining chatting capabilities. Experimental results show that ChatUIE has achieved significant performance improvements in information extraction tasks on multiple datasets.