Abstract:Large Language Models (LLMs) demonstrate impressive performance in diverse applications, yet they face significant drawbacks, including high inference latency, expensive training cost, and generation of hallucination. Collaborative decoding between large and small language models (SLMs) offers a novel approach to address these challenges. Inspired by dual-process cognitive theory, we integrate these methods into a unified framework termed Fast and Slow Generating (FS-GEN). This paper explores several techniques within the FS-GEN framework, including speculative decoding, contrastive decoding, and emulator or proxy fine-tuning. We provide a comprehensive analysis of these methodologies, offering insights into their similarities and differences under this framework. Our study delves into the differential knowledge capabilities of LLMs versus SLMs through the FS-GEN lens, revealing that fewer than 20% of collaborative interactions are required across various methods. These interactions adhere to a scaling law relative to the parameter ratios, thereby facilitating predictable collaboration. Furthermore, we investigate the specific positions where collaboration is most effective from an uncertainty perspective, yielding novel insights that could refine FS-GEN methods. Our findings reveal that the essential difference between models of different sizes lies in the uncertainty of the next token prediction, where interventions by larger models are most needed to assist the smaller ones. Code for Reproduction: <a class="link-external link-https" href="https://github.com/TsinghuaC3I/FS-GEN" rel="external noopener nofollow">this https URL</a>

From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models

A Survey on Efficient Inference for Large Language Models

A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding

Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems

Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models

The Efficiency Spectrum of Large Language Models: An Algorithmic Survey

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding

Quasi-random Multi-Sample Inference for Large Language Models

CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

Efficient Large Foundation Model Inference: A Perspective From Model and System Co-Design

Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens

Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations

Amortizing intractable inference in large language models

Inference acceleration for large language models using "stairs" assisted greedy generation

Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference

Graph-Structured Speculative Decoding

On Computing Paradigms - Where Will Large Language Models Be Going