Abstract:Large language models (LLMs) usually fall short on information extraction (IE) tasks and struggle to follow the complex instructions of IE tasks. This primarily arises from LLMs not being aligned with humans, as mainstream alignment datasets typically do not include IE data. In this paper, we introduce ADELIE (Aligning large language moDELs on Information Extraction), an aligned LLM that effectively solves various IE tasks, including closed IE, open IE, and on-demand IE. We first collect and construct a high-quality alignment corpus IEInstruct for IE. Then we train ADELIE_SFT using instruction tuning on IEInstruct. We further train ADELIE_SFT with direct preference optimization (DPO) objective, resulting in ADELIE_DPO. Extensive experiments on various held-out IE datasets demonstrate that our models (ADELIE_SFT and ADELIE_DPO) achieve state-of-the-art (SoTA) performance among open-source models. We further explore the general capabilities of ADELIE, and experimental results reveal that their general capabilities do not exhibit a noticeable decline. We will release the code, data, and models to facilitate further research.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address the shortcomings of large language models (LLMs) in information extraction (IE) tasks. Specifically, although LLMs have achieved significant results in many tasks after being aligned with human expectations (such as instruction fine-tuning or direct preference optimization), they still perform poorly in information extraction tasks, especially in closed information extraction tasks. The main reason is that existing mainstream alignment datasets usually do not include information extraction data, resulting in LLMs not being able to understand and follow complex IE task instructions well. To solve this problem, the authors propose ADELIE (Aligning large language mo DEL s on Information Extraction), a large-scale language model specifically aligned for information extraction tasks. ADELIE enhances LLMs' performance in IE tasks through the following two aspects: 1. **Rich Alignment Data**: A high-quality instruction tuning dataset IEInstruct was constructed, including 83,585 instances of various IE tasks. These instances cover a variety of instructions and input-output formats and are expanded through multiple augmentation techniques. 2. **Sufficient Alignment Training**: Based on the LLAMA 2 model, ADELIE is trained through supervised fine-tuning (SFT) and direct preference optimization (DPO), ensuring that the model excels in IE tasks while maintaining general capabilities. Experimental results show that ADELIE achieves state-of-the-art levels among open-source models in closed, open, and on-demand information extraction tasks, with no significant decline in general capabilities. Additionally, the authors analyze several key factors in the alignment process and provide valuable insights to promote future research.

ADELIE: Aligning Large Language Models on Information Extraction

ADELIE: Aligning Large Language Models on Information Extraction

AlignXIE: Improving Multilingual Information Extraction by Cross-Lingual Alignment

Diluie: Constructing Diverse Demonstrations of In-Context Learning with Large Language Model for Unified Information Extraction

Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction

Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment

CoLLiE: Collaborative Training of Large Language Models in an Efficient Way

Large Language Model Alignment: A Survey

Aligners: Decoupling LLMs and Alignment

Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!

Large Language Models for Generative Information Extraction: A Survey

MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

LIONs: An Empirically Optimized Approach to Align Language Models

GIELLM: Japanese General Information Extraction Large Language Model Utilizing Mutual Reinforcement Effect

REAL: Response Embedding-based Alignment for LLMs

DELIA: Diversity-Enhanced Learning for Instruction Adaptation in Large Language Models

Mixture-of-Instructions: Comprehensive Alignment of a Large Language Model through the Mixture of Diverse System Prompting Instructions

Aligning Large Language Models for Controllable Recommendations

InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning

Enabling and Analyzing How to Efficiently Extract Information from Hybrid Long Documents with LLMs