Abstract:The Large Language Models (LLMs) have been popular and widely used in creative ways because of their powerful capabilities. However, the substantial model size and complexity prevent LLMs from being implemented on resourceconstrained computing devices efficiently. Recent works that utilize quantization to boost the on-device efficiency of LLMs show that 8-bit or lower weight quantization is feasible with minimal impact on task performance with full-precision activation. Nevertheless, weight-only quantization does not leverage the potential acceleration on general edge processors, which typically support 16x16 and 8x8 integer multipliers. Additionally, it cannot benefit from specialized devices like Field-Programmable Gate Arrays (FPGAs), which offer reconfigure computing to design custom multipliers of arbitrary bit width. In this paper, we propose HotaQ, a Hardware-oriented token adaptive Quantization framework for general LLMs, and implement end-to-end accelerators on multiple devices. Considering the hardware profiling and activation analysis, we first introduce a basic activation quantization strategy to balance the trade-off of task performance and real inference speed. Then we integrate the activation-aware token pruning workflow to reduce the outliers and the adverse impact on attentivity. For mobile devices, we notably design the SIMD-based 4-bit multiplier and the efficient TRIP matrix multiplication for mobile devices. For FPGAs, we specially utilize the DSP packing techniques for 4/8-bit systolic-array-based multipliers and introduce an analytical model for achieving the best performance with limited on-chip resources. We apply our framework on different scales of LLMs, including LLaMA, OPT, and BLOOM with 4/8-bit for the activation and 4-bit for the weight. Experiments show that HotaQ achieves quantization on both model weights and activations while maintaining task performance comparable to existing weight-only quantization methods or even FP16 models. In the 4/8-bit scenario, HotaQ achieves an on-device speedup of up to 2.5× and 5.2× compared to its FP16 counterparts across edge devices and FPGAs respectively, marking a pioneering advancement in this domain.

Accuracy Evaluation of Long Short Term Memory Network Based Language Model with Fixed-Point Arithmetic.

HotaQ: Hardware Oriented Token Adaptive Quantization for Large Language Models

A Post-training Quantization Method for the Design of Fixed-Point-Based FPGA/ASIC Hardware Accelerators for LSTM/GRU Algorithms

C-LSTM: Enabling Efficient LSTM Using Structured Compression Techniques on FPGAs

A High Energy-Efficiency FPGA-Based LSTM Accelerator Architecture Design by Structured Pruning and Normalized Linear Quantization

FPGA Implementation of LSTM Based on Automatic Speech Recognition

FPGA Acceleration of Recurrent Neural Network Based Language Model

FPGA-based Accelerator for Long Short-Term Memory Recurrent Neural Networks

Efficient and Effective Sparse LSTM on FPGA with Bank-Balanced Sparsity

FPGA Acceleration of LSTM Based on Data for Test Flight.

FINN-L: Library Extensions and Design Trade-off Analysis for Variable Precision LSTM Networks on FPGAs

An Instruction-Driven Batch-Based High-Performance Resource-Efficient LSTM Accelerator on FPGA

A Comprehensive Evaluation of FPGA-Based Spatial Acceleration of LLMs

Implementation and Optimization of the Accelerator Based on FPGA Hardware for LSTM Network

Efficient Weight Reuse for Large LSTMs.

Laius: An 8-Bit Fixed-Point CNN Hardware Inference Engine

Acceleration of LSTM with Structured Pruning Method on FPGA

Error Models Of Finite Word Length Arithmetic In Cnn Accelerator Design

A Hardware Implementation of SNN-Based Spatio-Temporal Memory Model.

A Power-Efficient Accelerator Based on FPGAs for LSTM Network

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference