ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

Haoran Wu,Wenxuan Wang,Yuxuan Wan,Wenxiang Jiao,Michael Lyu

2023-03-15

Abstract:ChatGPT is a cutting-edge artificial intelligence language model developed by OpenAI, which has attracted a lot of attention due to its surprisingly strong ability in answering follow-up questions. In this report, we aim to evaluate ChatGPT on the Grammatical Error Correction(GEC) task, and compare it with commercial GEC product (e.g., Grammarly) and state-of-the-art models (e.g., GECToR). By testing on the CoNLL2014 benchmark dataset, we find that ChatGPT performs not as well as those baselines in terms of the automatic evaluation metrics (e.g., $F_{0.5}$ score), particularly on long sentences. We inspect the outputs and find that ChatGPT goes beyond one-by-one corrections. Specifically, it prefers to change the surface expression of certain phrases or sentence structure while maintaining grammatical correctness. Human evaluation quantitatively confirms this and suggests that ChatGPT produces less under-correction or mis-correction issues but more over-corrections. These results demonstrate that ChatGPT is severely under-estimated by the automatic evaluation metrics and could be a promising tool for GEC.

Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the issue of evaluating ChatGPT's performance on the Grammatical Error Correction (GEC) task and comparing it with commercial products (such as Grammarly) and current state-of-the-art models (such as GECToR). Specifically, the researchers aim to answer this question through the following points:

1. **How does ChatGPT perform on the GEC task?**
- Evaluate ChatGPT's performance on automatic evaluation metrics (such as F0.5 score) by testing on the CoNLL2014 benchmark dataset.

2. **What are the characteristics of ChatGPT's output?**
- Examine ChatGPT's output to understand whether it is limited to correcting errors one by one or if it makes more modifications to certain phrases or sentence structures while maintaining grammatical correctness.

3. **What are the results of human evaluation?**
- Quantify ChatGPT's performance in terms of under-correction, mis-correction, and over-correction through human evaluation to validate the limitations of automatic evaluation metrics.

4. **How does ChatGPT perform on sentences of different lengths?**
- Analyze the differences in ChatGPT's performance on short, medium-length, and long sentences, particularly whether its performance is worse on longer sentences.

Through this research, the authors aim to answer a core research question: **Is ChatGPT a good GEC tool?**

The research results indicate that although ChatGPT does not perform as well as baseline systems (such as GECToR and Grammarly) on automatic evaluation metrics, it performs well in human evaluations, especially in reducing under-correction and mis-correction. These results demonstrate the potential of ChatGPT in the GEC task while also highlighting the limitations of relying solely on automatic evaluation metrics.