Abstract:Although Large Language Models (LLMs) have demonstrated remarkable capabilities, their massive parameter counts and associated extensive computing make LLMs' deployment the main part of carbon emission from nowadays AI applications. Compared to modern GPUs like H$100$, it would be significantly carbon-sustainable if we could leverage old-fashioned GPUs such as M$40$ (as shown in Figure~\ref{fig:tisser}, M$40$ only has one third carbon emission of H$100$'s) for LLM servings. However, the limited High Bandwidth Memory (HBM) available on such GPU often cannot support the loading of LLMs due to the gigantic model size and intermediate activation data, making their serving challenging. For instance, a LLaMA2 model with $70$B parameters typically requires $128$GB for inference, which substantially surpasses $24$GB HBM in a $3090$ GPU and remains infeasible even considering the additional $64$GB DRAM. To address this challenge, this paper proposes a mixed-precision with a model modularization algorithm to enable LLM inference on outdated hardware with resource constraints. (The precision denotes the numerical precision like FP16, INT8, INT4) and multi-level caching (M2Cache).) Specifically, our M2Cache first modulizes neurons in LLM and creates their importance ranking. Then, it adopts a dynamic sparse mixed-precision quantization mechanism in weight space to reduce computational demands and communication overhead at each decoding step. It collectively lowers the operational carbon emissions associated with LLM inference. Moreover, M2Cache introduces a three-level cache management system with HBM, DRAM, and SSDs that complements the dynamic sparse mixed-precision inference. To enhance communication efficiency, M2Cache maintains a neuron-level mixed-precision LRU cache in HBM, a larger layer-aware cache in DRAM, and a full model in SSD.

RTiL: Real-Time Inference of Large Language Models on Memory-Constrained GPU Devices

High-throughput Generative Inference of Large Language Models with a Single GPU

Efficient LLM inference solution on Intel GPU

LLM in a flash: Efficient Large Language Model Inference with Limited Memory

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

Inference Performance Optimization for Large Language Models on CPUs

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference

Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU

Distributed Inference Performance Optimization for LLMs on CPUs

Efficient LLM Inference with I/O-Aware Partial KV Cache Recomputation

Efficient and Economic Large Language Model Inference with Attention Offloading

Practical offloading for fine-tuning LLM on commodity GPU via learned subspace projectors

InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference

Efficient Deployment of Large Language Model Across Cloud-Device Systems

CoLLM: A Collaborative LLM Inference Framework for Resource-Constrained Devices

Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching