Abstract:Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.

What problem does this paper attempt to address?

This paper mainly discusses the limitations of large-scale language models (LLMs) in information extraction (IE) tasks, even with the use of instance prompts, they cannot match the performance of small LMs fine-tuned on small-scale IE datasets. The authors observed that IE tasks such as named entity recognition (NER) and relation extraction (RE) focus on extracting important information, which can be formalized as a problem of matching labels to spans. Therefore, they proposed a new framework called MetaIE, which learns to extract "important information" from LLM through symbol distillation to construct a small LM as a meta-model, enabling this meta-model to adapt effectively to various IE tasks. MetaIE obtains a small LM through symbol distillation by following the scheme of matching labels to spans from LLM. Specifically, a distillation dataset is constructed by sampling sentences from the language model pretraining dataset and prompting LLM to recognize typed spans of "important information". The experimental evaluation assesses the performance of the meta-model in a few-shot adaptation setting and compares it with other methods, including original language model pretraining, pretraining with multiple IE tasks annotated by humans, and symbol distillation of LLM for a single IE task. The results show that MetaIE outperforms other methods in multiple IE tasks, especially in tasks that multi-task pretraining methods cannot handle. In addition, the paper provides a comprehensive analysis of MetaIE, studying the impact of factors such as distillation dataset size, meta-model architecture, and meta-model size. The success of MetaIE is not only attributed to LLMs but also to the scheme of matching labels to spans proposed by the authors. In conclusion, this paper addresses the problem of how to leverage the knowledge of LLMs to create a meta-model that can efficiently adapt to various information extraction tasks. Through symbol distillation, their proposed method, MetaIE, exhibits better performance than other methods with limited samples.

MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

Universal Information Extraction with Meta-Pretrained Self-Retrieval

Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractors

Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!

ADELIE: Aligning Large Language Models on Information Extraction

Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment

A Meta Learning Approach for Open Information Extraction

MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

Rethinking Meta-Learning from a Learning Lens

Meta-Models: An Architecture for Decoding LLM Behaviors Through Interpreted Embeddings and Natural Language

CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning

Effective Few-Shot Named Entity Linking by Meta-Learning

Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction

AlignXIE: Improving Multilingual Information Extraction by Cross-Lingual Alignment

Diluie: Constructing Diverse Demonstrations of In-Context Learning with Large Language Model for Unified Information Extraction

Gradient Imitation Reinforcement Learning for General Low-Resource Information Extraction

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

Gradient Imitation Reinforcement Learning for General Low-Resource Information Extraction.