Abstract:Universal information extraction (Universal IE) aims to develop one model capable of solving multiple IE target tasks. Previous works have enhanced extraction performance of target tasks through auxiliary tasks. However, there are still limitations in terms of learning strategies. From one aspect, joint learning-based universal IE approaches, which simply mix auxiliary tasks with target tasks, fail to enable the model to master basic knowledge from auxiliary tasks before learning target tasks. From another aspect, continual learning-based universal IE approaches, which sequentially update all the model parameters on auxiliary tasks and target tasks, tend to cause catastrophic forgetting. In this study, we design a multi-LoRA continual learning-based instruction fine-tuning framework for universal IE. Specifically, we design unique LoRA modules for learning auxiliary tasks and target tasks. We first freeze pre-trained weights and update additional parameters on auxiliary tasks through one LoRA module. Subsequently, we keep the weights frozen and further adjust parameters through another LoRA module to adapt the model to the target tasks. Finally, we merge the frozen weights with learned weights, thereby enabling the model to better leverage the acquired abilities during the inference phase. Therefore, our model masters basic extraction abilities before learning target tasks and does not forget this basic knowledge during the target learning process. Moreover, we regard extraction, classification, and recognition as basic abilities and further design auxiliary tasks based on these basic abilities. Experimental results on 37 datasets across 3 tasks show that our approach reaches state-of-the-art performance.

MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction

Multi-LoRA Continual Learning Based Instruction Tuning Framework for Universal Information Extraction

CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment

S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method-Tuning: A Simple Cross-lingual Sub-network Tuning Method

AlignXIE: Improving Multilingual Information Extraction by Cross-Lingual Alignment

INFOXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Tasks

Translation and Fusion Improves Zero-shot Cross-lingual Information Extraction

xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning

milIE: Modular & Iterative Multilingual Open Information Extraction

Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment

MMM: Multilingual Mutual Reinforcement Effect Mix Datasets & Test with Open-domain Information Extraction Large Language Models

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction.

Crosslingual Generalization through Multitask Finetuning

Cross-model Control: Improving Multiple Large Language Models in One-time Training

X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions

Enhancing Model Performance in Multilingual Information Retrieval with Comprehensive Data Engineering Techniques

Joint Information Extraction with Cross-Task and Cross-Instance High-Order Modeling

XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation