Abstract:Quantization has been substantially adopted to accelerate inference and reduce memory consumption of large language models (LLMs). While activation-weight joint quantization speeds up the inference process through low-precision kernels, we demonstrate that it suffers severe performance degradation on multi-step reasoning tasks, rendering it ineffective. We propose a novel quantization paradigm called QSPEC, which seamlessly integrates two complementary quantization schemes for speculative decoding. Leveraging nearly cost-free execution switching, QSPEC drafts tokens with low-precision, fast activation-weight quantization, and verifies them with high-precision weight-only quantization, effectively combining the strengths of both quantization schemes. Compared to high-precision quantization methods, QSPEC empirically boosts token generation throughput by up to 1.80x without any quality compromise, distinguishing it from other low-precision quantization approaches. This enhancement is also consistent across various serving tasks, model sizes, quantization methods, and batch sizes. Unlike existing speculative decoding techniques, our approach reuses weights and the KV cache, avoiding additional memory overhead. Furthermore, QSPEC offers a plug-and-play advantage without requiring any training. We believe that QSPEC demonstrates unique strengths for future deployment of high-fidelity quantization schemes, particularly in memory-constrained scenarios (e.g., edge devices).

Low Complexity State Metric Memory Reduction for Turbo Decoding with Stochastic Quantization

Reduced Memory Decoding Schemes for Turbo Decoding Based on Storing the Index of the State Metric.

Joint States Dimension Reduction and Quantizer Design for Estimation under Communication Constraints

A state metrics compressed decoding technique for energy-efficient turbo decoder

Low Complexity State Metric Compression Technique In Turbo Decoder

A low-memory intensive decoding architecture for double-binary convolutional turbo code

High efficient STDMSD rate control method

Nonlinear Quantization for In-Sram Multi-Bit MAC Design

An Anti-Overflow Control Method of State Metrics for High Throughput Turbo Decoder

Reverse Calculation-Based Low Memory Turbo Decoder for Power Constrained Applications

A Memory Reduced Decoding Scheme for Double Binary Convolutional Turbo Code Based on Forward Recalculation

Study on Memory Efficient LDPC Decoding Using Dynamic Quantization

Quantization Issues in Turbo Decoding

Stochastic Hybrid Combining Design For Quantized Massive Mimo Systems

QSpec: Speculative Decoding with Complementary Quantization Schemes

Efficient Quantization Algorithm for Low-Density Parity-Check Code Decoding

A Novel Decoding Scheme Based On Recalculation For Double Binary Convolutional Turbo Code

High speed Turbo decoder design based on stochastic computation

A Memory Reduced Turbo Code Decoding Architecture for LTE-Advanced Standard Based on Reverse Recalculation

Memory-Reduced Map Decoding for Double-Binary Convolutional Turbo Code

Memory Efficient LDPC Decoder Design