Abstract:Large language models (LLMs) suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically, billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation. Recently, parallel decoding, a type of speculative decoding algorithms, is becoming more popular and has demonstrated impressive efficiency improvement in generation. It introduces extra decoding heads to large models, enabling them to predict multiple subsequent tokens simultaneously and verify these candidate continuations in a single decoding step. However, this approach deviates from the training objective of next token prediction used during pre-training, resulting in a low hit rate for candidate tokens. In this paper, we propose a new speculative decoding algorithm, Clover, which integrates sequential knowledge into the parallel decoding process. This enhancement improves the hit rate of speculators and thus boosts the overall efficiency. Clover transmits the sequential knowledge from pre-speculated tokens via the Regressive Connection, then employs an Attention Decoder to integrate these speculated tokens. Additionally, Clover incorporates an Augmenting Block that modifies the hidden states to better align with the purpose of speculative generation rather than next token prediction. The experiment results demonstrate that Clover outperforms the baseline by up to 91% on Baichuan-Small and 146% on Baichuan-Large, respectively, and exceeds the performance of the previously top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on Baichuan-Large, respectively.

Context-aware Code Generation with Synchronous Bidirectional Decoder

Learning to Decode for Future Success

A Simple, Fast Diverse Decoding Algorithm for Neural Generation

From Code to Natural Language: Type-Aware Sketch-Based Seq2Seq Learning

CoreGen: Contextualized Code Representation Learning for Commit Message Generation

Fast Contextual Scene Graph Generation with Unbiased Context Augmentation.

A new approach for encoding code and assisting code understanding

A Branching Decoder for Set Generation

Decoder : Generate rewritten-utterance Encoder : Last Utterance RepresentationEncoder : Context Representation

Code generation from a graphical user interface via attention-based encoder–decoder model

Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge

Toward Exploring the Code Understanding Capabilities of Pre-trained Code Generation Models

Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass

Pseudo-Bidirectional Decoding for Local Sequence Transduction

Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generation

A Grammar-Based Structural CNN Decoder for Code Generation

A Framework for Bidirectional Decoding: Case Study in Morphological Inflection

Code Search based on Context-aware Code Translation

CodeGRU: Context-aware deep learning with gated recurrent unit for source code modeling

Context Perception Parallel Decoder for Scene Text Recognition