On Speculative Decoding for Multimodal Large Language Models

Mukul Gagrani,Raghavv Goel,Wonseok Jeon,Junyoung Park,Mingu Lee,Christopher Lott
2024-04-13
Abstract:Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37$\times$ using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.
Computation and Language,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
This paper discusses how to improve the efficiency of inference for multimodal large language models (MLLMs). Due to the memory bandwidth bottleneck and the constraints of autoregressive generation, researchers proposed a speculative decoding method to accelerate the inference of the LLaVA 7B model. They found that using only the language model as a draft model can effectively perform speculative decoding without the need for image tokens and their associated processing components, thereby improving speed. By training a language model with 115M parameters, the experiments showed that this method can achieve a maximum speed improvement of 2.37x under memory constraints in three different tasks. In addition, they also constructed a compact LLaVA draft model with image adapters, which showed slight performance improvement in image caption generation but performed comparably to the pure language model draft model in other tasks. Overall, the paper demonstrates the potential of speculative decoding in multimodal language models and provides directions for further optimization.