Abstract:Large language models (LLMs) have achieved remarkable success across diverse tasks, yet their inference processes are hindered by substantial time and energy demands due to single-token generation at each decoding step. While previous methods such as speculative decoding mitigate these inefficiencies by producing multiple tokens per step, each token is still generated by its single-token distribution, thereby enhancing speed without improving effectiveness. In contrast, our work simultaneously enhances inference speed and improves the output effectiveness. We consider multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration, theoretically reducing perplexity and enhancing task performance. However, MTJD suffers from the high cost of sampling from the joint distribution of multiple tokens. Inspired by speculative decoding, we introduce multi-token assisted decoding (MTAD), a novel framework designed to accelerate MTJD. MTAD leverages a smaller auxiliary model to approximate the joint distribution of a larger model, incorporating a verification mechanism that not only ensures the accuracy of this approximation, but also improves the decoding efficiency over conventional speculative decoding. Theoretically, we demonstrate that MTAD closely approximates exact MTJD with bounded error. Empirical evaluations using Llama-2 and OPT models ranging from 13B to 70B parameters across various tasks reveal that MTAD reduces perplexity by 21.2% and improves downstream performance compared to standard single-token sampling. Furthermore, MTAD achieves a 1.42x speed-up and consumes 1.54x less energy than conventional speculative decoding methods. These results highlight MTAD's ability to make multi-token joint decoding both effective and efficient, promoting more sustainable and high-performance deployment of LLMs.

Joint Decoding with Multiple Translation Models.

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Joint Parsing and Translation

Joint Decoding of Tree Transduction Models for Sentence Compression

Joint Tokenization and Translation

Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding

Flexible and Efficient Hypergraph Interactions for Joint Hierarchical and Forest-to-String Decoding.

Better Simultaneous Translation with Monotonic Knowledge Distillation.

4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders

Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation

C L ] 1 0 Ju n 20 18 Deconvolution-Based Global Decoding for Neural Machine Translation

Modeling Voting for System Combination in Machine Translation

Plug, Play, and Fuse: Zero-Shot Joint Decoding via Word-Level Re-ranking Across Diverse Vocabularies

Deconvolution-Based Global Decoding for Neural Machine Translation.

Lattice-based System Combination for Statistical Machine Translation.

Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference

Joint Training for Neural Machine Translation Models with Monolingual Data

Learning to Decode Collaboratively with Multiple Language Models

Deep Learning Model-Based Machine Learning for Chinese and Japanese Translation

Joint Training for Pivot-based Neural Machine Translation.

Joint tokenization, parsing, and translation