Abstract:The widely-used, weight-only quantized large language models (LLMs), which leverage low-bit integer (INT) weights and retain floating-point (FP) activations, reduce storage requirements while maintaining accuracy. However, this shifts the energy and latency bottlenecks towards the FP activations that are associated with costly memory accesses and computations. Existing LLM accelerators focus primarily on computation optimizations, overlooking the potential of jointly optimizing FP computations and data movement, particularly for the dominant FP-INT GeMM operations in LLM inference. To address these challenges, we investigate the sensitivity of activation precision across various LLM modules and its impact on overall model accuracy. Based on our findings, we first propose the Anda data type: an adaptive data format with group-shared exponent bits and dynamic mantissa bit allocation. Secondly, we develop an iterative post-training adaptive precision search algorithm that optimizes the bit-width for different LLM modules to balance model accuracy, energy efficiency, and inference speed. Lastly, a suite of hardware optimization techniques is proposed to maximally exploit the benefits of the Anda format. These include a bit-plane-based data organization scheme, Anda-enhanced processing units with bit-serial computation, and a runtime bit-plane Anda compressor to simultaneously optimize storage, computation, and memory footprints. Our evaluations on FPINT GeMM operations show that Anda achieves a 2.4x speedup, 4.0x area efficiency, and 3.1x energy efficiency improvement on average for popular LLMs including OPT, LLaMA, and LLaMA-2 series over the GPU-like FP-FP baseline. Anda demonstrates strong adaptability across various application scenarios, accuracy requirements, and system performance, enabling efficient LLM inference across a wide range of deployment scenarios.

Accelerating Generalized Linear Models with MLWeaving

Accelerating Generalized Linear Models with MLWeaving: A One-Size-Fits-All System for Any-precision Learning

MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning

Doppiodb 2.0

Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization

Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Accelerators

Progressive Mixed-Precision Decoding for Efficient LLM Inference

Efficient Construction of Nonlinear Models over Normalized Data

A Versatile Acceleration Framework for Machine Learning Algorithms

Accelerating Production LLMs with Combined Token/Embedding Speculators

LLM-Enhanced Data Management

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

Optimizing LLM Queries in Relational Workloads

Accelerating Generalized Linear Models by Trading off Computation for Uncertainty

Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs

Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines