Abstract:Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, autoregressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as grammatical error correction, text summarization, text style transfer, dialogue, semantic parsing, automatic speech recognition, and so on. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, reasonable training objectives, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications.

A survey on non-autoregressive generation for neural machine translation and beyond

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Hybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation

A Study of Non-autoregressive Model for Sequence Generation

Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

Non-Autoregressive Machine Translation with Auxiliary Regularization

Neural-machine-translation-based Commit Message Generation: How Far Are We?

Non-Autoregressive Document-Level Machine Translation

What Have We Achieved on Non-autoregressive Translation?

Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information

A Survey on Low-Resource Neural Machine Translation

Improving Non-autoregressive Neural Machine Translation with Monolingual Data

Imitation Learning for Non-Autoregressive Neural Machine Translation.

Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation

Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints

Glancing Transformer for Non-Autoregressive Neural Machine Translation.

Semi-Autoregressive Neural Machine Translation

Diffusion Models for Non-autoregressive Text Generation: A Survey

Unlocking the Power of GANs in Non-Autoregressive Text Generation

Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation

Neural Machine Translation Advised by Statistical Machine Translation