Abstract:Large language models (LLMs) have demonstrated remarkable capabilities in tasks requiring reasoning and multi-step problem-solving through the use of chain-of-thought (CoT) prompting. However, generating the full CoT process results in significantly longer output sequences, leading to increased computational costs and latency during inference. To address this challenge, we propose a novel approach to compress the CoT process through semantic alignment, enabling more efficient decoding while preserving the benefits of CoT reasoning. Our method introduces an auxiliary CoT model that learns to generate and compress the full thought process into a compact special token representation semantically aligned with the original CoT output. This compressed representation is then integrated into the input of the Hidden Chain-of-Thought (HCoT) model. The training process follows a two-stage procedure: First, the CoT model is optimized to generate the compressed token representations aligned with the ground-truth CoT outputs using a contrastive loss. Subsequently, with the CoT model parameters frozen, the HCoT model is fine-tuned to generate accurate subsequent predictions conditioned on the prefix instruction and the compressed CoT representations from the CoT model. Extensive experiments across three challenging domains - mathematical reasoning, agent invocation, and question answering - demonstrate that our semantic compression approach achieves competitive or improved performance compared to the full CoT baseline, while providing significant speedups of at least 1.5x in decoding time. Moreover, incorporating contrastive learning objectives further enhances the quality of the compressed representations, leading to better CoT prompting and improved task accuracy. Our work paves the way for more efficient exploitation of multi-step reasoning capabilities in LLMs across a wide range of applications.

Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster

Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs

Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding

Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference

Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration

Deliberation in Latent Space via Differentiable Cache Augmentation

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

Compressed Chain of Thought: Efficient Reasoning Through Dense Representations

Accelerating Transformer Inference for Translation via Parallel Decoding

Fast Decoding in Sequence Models using Discrete Latent Variables

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

SPEED: Speculative Pipelined Execution for Efficient Decoding

Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement

TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation

FIRP: Faster LLM inference via future intermediate representation prediction

SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Training-Free Exponential Context Extension via Cascading KV Cache

Tandem Transformers for Inference Efficient LLMs