RedPenNet for Grammatical Error Correction: Outputs to Tokens, Attentions to Spans

Bohdan Didenko,Andrii Sameliuk

2023-09-20

Abstract:The text editing tasks, including sentence fusion, sentence splitting and rephrasing, text simplification, and Grammatical Error Correction (GEC), share a common trait of dealing with highly similar input and output sequences. This area of research lies at the intersection of two well-established fields: (i) fully autoregressive sequence-to-sequence approaches commonly used in tasks like Neural Machine Translation (NMT) and (ii) sequence tagging techniques commonly used to address tasks such as Part-of-speech tagging, Named-entity recognition (NER), and similar. In the pursuit of a balanced architecture, researchers have come up with numerous imaginative and unconventional solutions, which we're discussing in the Related Works section. Our approach to addressing text editing tasks is called RedPenNet and is aimed at reducing architectural and parametric redundancies presented in specific Sequence-To-Edits models, preserving their semi-autoregressive advantages. Our models achieve $F_{0.5}$ scores of 77.60 on the BEA-2019 (test), which can be considered as state-of-the-art the only exception for system combination and 67.71 on the UAGEC+Fluency (test) benchmarks. This research is being conducted in the context of the UNLP 2023 workshop, where it was presented as a paper as a paper for the Shared Task in Grammatical Error Correction (GEC) for Ukrainian. This study aims to apply the RedPenNet approach to address the GEC problem in the Ukrainian language.

Computation and Language

What problem does this paper attempt to address?

The paper primarily focuses on addressing the problem of Grammatical Error Correction (GEC), particularly for the Ukrainian language. Specifically, the researchers propose a new architecture called RedPenNet, which aims to reduce the architectural and parameter redundancy in existing Sequence-To-Edits models while retaining their semi-autoregressive advantages. RedPenNet corrects input text by generating a series of edit operations and can handle various text editing tasks, including sentence fusion, splitting, rewriting, simplification, and grammatical error correction.

The core contributions of the paper are:
1. **Proposing the RedPenNet model**: This is an Insertion-Only Autoregressive (IOAR) model, a subclass of Sequence-To-Edits, capable of determining edit spans while generating edit tokens.
2. **Efficient decoder vocabulary**: To enhance efficiency, the paper trains a compact task-specific decoder BPE vocabulary to reduce the cost of pre-softmax dot operations.
3. **Multilingual support**: By using a multilingual pre-trained encoder and language-specific decoders, RedPenNet can handle GEC tasks in multiple languages.
4. **Experimental validation**: In the UNLP 2023 shared task, RedPenNet achieved excellent results in the Ukrainian GEC+Fluency benchmark, demonstrating its effectiveness.

Through these improvements, RedPenNet not only enhances the performance of GEC tasks but also shows potential in handling text editing tasks across different languages.