Abstract:In this study, we investigate in-context learning (ICL) in document-level event argument extraction (EAE) to alleviate the dependency on large-scale labeled data for this task. We introduce the Heuristic-Driven Link-of-Analogy (HD-LoA) prompting to address the challenge of example selection and to develop a prompting strategy tailored for EAE. Specifically, we hypothesize and validate that LLMs learn task-specific heuristics from demonstrations via ICL. Building upon this hypothesis, we introduce an explicit heuristic-driven demonstration construction approach, which transforms the haphazard example selection process into a methodical method that emphasizes task heuristics. Additionally, inspired by the analogical reasoning of human, we propose the link-of-analogy prompting, which enables LLMs to process new situations by drawing analogies to known situations, enhancing their performance on unseen classes beyond limited ICL examples. Experiments show that our method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets. Additionally, the HD-LoA prompting shows effectiveness in diverse tasks like sentiment analysis and natural language inference, demonstrating its broad adaptability.
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve
This paper aims to address the dependency on large-scale annotated data in the task of Document-Level Event Argument Extraction (EAE). Specifically, the authors introduce a Heuristic-Driven Link-of-Analogy (HD-LoA) prompting strategy to reduce the need for large-scale annotated data and improve model performance in low-resource scenarios.
### Background and Motivation
1. **Challenges of Document-Level Event Argument Extraction**:
- The goal of document-level event argument extraction is to extract structured event argument information from documents, which is crucial for many applications.
- Current methods mainly rely on large-scale annotated data and supervised learning, which is not only costly but also particularly challenging for complex tasks.
2. **Advantages of In-Context Learning (ICL)**:
- ICL is an emerging capability of large language models (LLMs) that guides the model to complete tasks using a few input-output examples, thereby reducing the dependency on large-scale annotated data.
- However, the application of ICL in document-level event argument extraction faces several challenges, including example selection, context length limitations, event type diversity, and prompting strategies for non-inference tasks.
### Main Contributions
1. **Hypothesis and Validation**:
- The authors hypothesize that LLMs can learn task-specific heuristic rules from examples and validate this hypothesis through experiments.
- Experimental results show that successful prompts usually contain multiple heuristic rules, and the diversity of heuristic rules significantly impacts ICL performance.
2. **Heuristic-Driven Example Construction Method**:
- A method is proposed to explicitly incorporate task heuristic rules into examples, transforming the random example selection process into a systematic approach that emphasizes the diversity of task heuristics.
- This method helps to explicitly check whether examples contain diverse heuristic rules and ensures that LLMs can recognize these rules.
3. **Link-of-Analogy Prompting Strategy**:
- To address the issue of event type diversity, a link-of-analogy prompting strategy is proposed, which handles new situations by analogy to known ones, enhancing the model's ability to deal with unseen event types.
- Experimental results show that this strategy performs well in handling unseen event types.
4. **Broad Adaptability**:
- Besides its application in document-level event argument extraction, the HD-LoA prompting strategy also achieves significant performance improvements in sentiment analysis and natural language inference tasks, demonstrating its broad adaptability.
### Experimental Results
1. **Document-Level Event Argument Extraction Task**:
- On the RAMS and DocEE datasets, the HD-LoA prompting strategy significantly outperforms existing prompting methods and few-shot supervised learning methods, especially in the event argument classification (Arg-C) task, with F1 scores improved by 4.65%, 3.41%, and 6.03%, respectively.
2. **Sentiment Analysis and Natural Language Inference Tasks**:
- On the SST-2 and SNLI datasets, the HD-LoA prompting strategy also shows significant performance improvements, with accuracy increased by 2.87% and 2.63%, respectively.
3. **Comparison with Fully Trained Supervised Models**:
- Although fully trained supervised models usually have higher accuracy after training on a large number of samples, the HD-LoA prompting strategy even surpasses these models on the cross-domain setting of the DocEE dataset.
### Conclusion
By introducing the heuristic-driven link-of-analogy prompting strategy, this paper effectively addresses the dependency on large-scale annotated data in the document-level event argument extraction task and demonstrates its broad application potential across multiple tasks.