Abstract:As Large Language Models (LLMs) have made significant advancements across various tasks, such as question answering, translation, text summarization, and dialogue systems, the need for accuracy in information becomes crucial, especially for serious financial products serving billions of users like Alipay. However, for a real-world product serving millions of users, the inference speed of LLMs becomes a critical factor compared to a mere experimental model. Hence, this paper presents a generic framework for accelerating the inference process, resulting in a substantial increase in speed and cost reduction for our LLM-based scenarios, with lossless generation accuracy. In the traditional inference process, each token is generated sequentially by the LLM, leading to a time consumption proportional to the number of generated tokens. To enhance this process, our framework, named \textit{lookahead}, introduces a \textit{multi-branch} strategy. Instead of generating a single token at a time, we propose a Trie-based retrieval and verification mechanism to be able to accept several tokens at a forward step. Our strategy offers two distinct advantages: (1) it guarantees absolute correctness of the output, avoiding any approximation algorithms, and (2) the worst-case performance of our approach is equivalent to the conventional process. We conduct extensive experiments to demonstrate the significant improvements achieved by applying our inference acceleration framework. Our framework is widely deployed in Alipay since April 2023, and obtain remarkable 2.66x to 6.26x speedup. Our code is available at <a class="link-external link-https" href="https://github.com/alipay/PainlessInferenceAcceleration" rel="external noopener nofollow">this https URL</a>.

Reducing time-synchronous beam search effort using stage based look-ahead and language model rank based pruning

An Efficient Layer-Wised Beam Pruning Algorithm for Large Vocabulary Continuous Speech Recognition System

Voice-based Local Search Using a Language Model Look-ahead Structure for Efficient Pruning

Fast Language Model Look-ahead Algorithm Using Extended N -Gram Model

A Novel Beam Search Algorithm Of Speech Recognition For Voice Command Control

Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition

A study of modifying pruning strategies for dp beam search at a preset input frame

Just-in-time Latent Semantic Adaptation on Language Model for Chinese Speech Recognition Using Web Data

Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model

A Transcription Prompt-based Efficient Audio Large Language Model for Robust Speech Recognition

A two-layer lexical tree based beam search in continuous Chinese speech recognition

An Online Attention-based Model for Speech Recognition

Efficient Embedded Speech Recognition for Very Large Vocabulary Mandarin Car-Navigation Systems

Multi-stage Large Language Model Correction for Speech Recognition

Using fine-tuning and min lookahead beam search to improve Whisper

Leveraging Large Language Models for Exploiting ASR Uncertainty

Discriminative Pruning of Language Models for Chinese Word Segmentation

Faster Speech-LLaMA Inference with Multi-token Prediction

Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model

Low Latency ASR for Simultaneous Speech Translation

Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy