Abstract:The recent emergence of Large Language Models based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these models have long inference latency, which limits their deployment and makes them prohibitively expensive for various real-time applications. The inference latency is further exacerbated by autoregressive generative tasks, as models need to run iteratively to generate tokens sequentially without leveraging token-level parallelization. To address this, we propose Big Little Decoder (BiLD), a framework that can improve inference efficiency and latency for a wide range of text generation applications. The BiLD framework contains two models with different sizes that collaboratively generate text. The small model runs autoregressively to generate text with a low inference cost, and the large model is only invoked occasionally to refine the small model's inaccurate predictions in a non-autoregressive manner. To coordinate the small and large models, BiLD introduces two simple yet effective policies: (1) the fallback policy that determines when to hand control over to the large model; and (2) the rollback policy that determines when the large model needs to correct the small model's inaccurate predictions. To evaluate our framework across different tasks and models, we apply BiLD to various text generation scenarios encompassing machine translation on IWSLT 2017 De-En and WMT 2014 De-En, and summarization on XSUM and CNN/DailyMail. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Furthermore, our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture. Our code is open-sourced

InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Model

Emage: Non-Autoregressive Text-to-Image Generation

Insertion-based Decoding with automatically Inferred Generation Order

PinNet: Pinpoint Instructive Information for Retrieval Augmented Code-to-Text Generation

POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise

Tigs: An Inference Algorithm For Text Infilling With Gradient Search

Dynamic and Efficient Inference for Text Generation Via BERT Family

Non-iterative Parallel Text Generation via Glancing Transformer

Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap Between Training and Inference!

Semi-Autoregressive Neural Machine Translation

An Encoder with non-Sequential Dependency for Neural Data-to-Text Generation.

Self-Infilling Code Generation

Towards Incremental NER Data Augmentation Via Syntactic-aware Insertion Transformer.

Non-Autoregressive Text Generation with Pre-trained Language Models

Instant3D: Instant Text-to-3D Generation

Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference

Improving Open-Ended Text Generation via Adaptive Decoding

Speculative Decoding with Big Little Decoder

Cascaded Text Generation with Markov Transformers