Abstract:Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, autoregressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as grammatical error correction, text summarization, text style transfer, dialogue, semantic parsing, automatic speech recognition, and so on. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, reasonable training objectives, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications.

JANUS: Joint Autoregressive and Non-autoregressive Training with Auxiliary Loss for Sequence Generation

Hybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation

Hybrid Autoregressive and Non-Autoregressive Transformer Models for Speech Recognition

Unlocking the Power of GANs in Non-Autoregressive Text Generation

Attentive Multi-Layer Perceptron for Non-autoregressive Generation

Incorporating BERT into Parallel Sequence Decoding with Adapters.

Latent-Glat: Glancing at Latent Variables for Parallel Text Generation

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

A Study of Non-autoregressive Model for Sequence Generation

Fast Sequence Generation with Multi-Agent Reinforcement Learning

PNAT: Non-autoregressive Transformer by Position Learning

Hint-Based Training for Non-Autoregressive Machine Translation.

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining

Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation

Non-iterative Parallel Text Generation via Glancing Transformer

APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

On the Learning of Non-Autoregressive Transformers.

Semi-Autoregressive Neural Machine Translation

Jointly Masked Sequence-To-Sequence Model For Non-Autoregressive Neural Machine Translation

JumpCoder: Go Beyond Autoregressive Coder via Online Modification