Abstract:Instruction fine-tuned language models on a collection of instruction annotated datasets (FLAN) have shown highly effective to improve model performance and generalization to unseen tasks. However, a majority of standard parsing tasks including abstract meaning representation (AMR), universal dependency (UD), semantic role labeling (SRL) has been excluded from the FLAN collections for both model training and evaluations. In this paper, we take one of such instruction fine-tuned pre-trained language models, i.e. FLAN-T5, and fine-tune them for AMR parsing. Our extensive experiments on various AMR parsing tasks including AMR2.0, AMR3.0 and BioAMR indicate that FLAN-T5 fine-tuned models out-perform previous state-of-the-art models across all tasks. In addition, full fine-tuning followed by the parameter efficient fine-tuning, LoRA, further improves the model performances, setting new state-of-the-arts in Smatch on AMR2.0 (86.4), AMR3.0 (84.9) and BioAMR (82.3).

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in standard parsing tasks, especially in Abstract Meaning Representation (AMR) parsing tasks, the effectiveness of instruction - tuned language models has not been fully verified. Although instruction - tuned language models perform well in most natural language processing tasks, standard parsing tasks such as AMR parsing, Universal Dependencies (UD) and Semantic Role Labeling (SRL) are usually excluded from the fine - tuning data set, whether for model training or evaluation. Therefore, the goal of this paper is to explore and verify the application effect of instruction - tuned language models (specifically the FLAN - T5 model) in AMR parsing tasks, in order to improve model performance and promote technological progress in these fields. The main contributions of the paper include: 1. **First application**: For the first time, the instruction - tuned FLAN - T5 model is applied to AMR parsing tasks, showing significant improvements of these models in AMR parsing over the previous best model based on BART fine - tuning. 2. **Parameter - efficient fine - tuning technique**: The application of LoRA (Low - Rank Adaptation), a parameter - efficient fine - tuning technique, in sequence - to - sequence tasks is explored. Although the performance of the model fine - tuned only with LoRA is not as good as that of the fully fine - tuned model, applying LoRA fine - tuning to the fully fine - tuned model can further improve the model performance. 3. **New best performance**: Through the above methods, the paper sets new best performance records on AMR2.0, AMR3.0 and BioAMR tasks, with Smatch scores of 86.4, 84.9 and 82.3 respectively. In conclusion, this paper verifies the effectiveness and potential of instruction - tuned language models in AMR parsing tasks through experiments, providing new directions and technical support for research in this field.

AMR Parsing with Instruction Fine-tuned Pre-trained Language Models

Pushing the Limits of AMR Parsing with Self-Learning

XLPT-AMR: Cross-Lingual Pre-Training via Multi-Task Learning for Zero-Shot AMR Parsing and Text Generation

Improving AMR Parsing with Sequence-to-Sequence Pre-training

ATP: AMRize then Parse! Enhancing AMR Parsing with PseudoAMRs

Making Better Use of Bilingual Information for Cross-Lingual AMR Parsing.

Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing

Guiding AMR Parsing with Reverse Graph Linearization

AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing

Integrating Syntactic and Semantic Knowledge in AMR Parsing with Heterogeneous Graph Attention Network

Exploration and Comparison of Diverse Approaches for Integrating Syntactic Knowledge into AMR Parsing

Hierarchical Curriculum Learning for AMR Parsing

Hierarchical information matters! Improving AMR parsing with multi-granularity representation interactions

Scaling Instruction-Finetuned Language Models

Should Cross-Lingual AMR Parsing go Meta? An Empirical Assessment of Meta-Learning and Joint Learning AMR Parsing

Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuning

AMR Parsing as Sequence-to-Graph Transduction

Stacked AMR Parsing with Silver Data

Automatic Accuracy Prediction for AMR Parsing

AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

Addressing Long-Distance Dependencies in AMR Parsing with Hierarchical Clause Annotation