Abstract:Large language model (LLM) decoding involves generating a sequence of tokens based on a given context, where each token is predicted one at a time using the model's learned probabilities. The typical autoregressive decoding method requires a separate forward pass through the model for each token generated, which is computationally inefficient and poses challenges for deploying LLMs in latency-sensitive scenarios. The main limitations of current decoding methods stem from their inefficiencies and resource demands. Existing approaches either necessitate fine-tuning smaller models, which is resource-intensive, or rely on fixed retrieval schemes to construct drafts for the next tokens, which lack adaptability and fail to generalize across different models and contexts. To address these issues, we introduce a novel methodology called ADED, which accelerates LLM decoding without requiring fine-tuning. Our approach involves an adaptive draft-verification process that evolves over time to improve efficiency. We utilize a tri-gram matrix-based LLM representation to dynamically approximate the output distribution of the LLM, allowing the model to adjust to changing token probabilities during the decoding process. Additionally, we implement a draft construction mechanism that effectively balances exploration and exploitation, ensuring that the drafts generated are both diverse and close to the true output distribution of the LLM. The importance of this design lies in its ability to optimize the draft distribution adaptively, leading to faster and more accurate decoding. Through extensive experiments on various benchmark datasets and LLM architectures, we demonstrate that ADED significantly accelerates the decoding process while maintaining high accuracy, making it suitable for deployment in a wide range of practical applications.

Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access

Graph-Structured Speculative Decoding

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding

Speculative Contrastive Decoding

Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding

Grammar-Aligned Decoding

Adaptive Draft-Verification for Efficient Large Language Model Decoding

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

On Speculative Decoding for Multimodal Large Language Models

Adaptive Skeleton Graph Decoding

Accelerating LLM Inference with Staged Speculative Decoding

Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher

SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models

Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding

SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens

BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models

S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models

Control Large Language Models via Divide and Conquer

Investigating Decoder-only Large Language Models for Speech-to-text Translation

Dynamic-Width Speculative Beam Decoding for Efficient LLM Inference