Abstract:The Large Language Models (LLMs) have been popular and widely used in creative ways because of their powerful capabilities. However, the substantial model size and complexity prevent LLMs from being implemented on resourceconstrained computing devices efficiently. Recent works that utilize quantization to boost the on-device efficiency of LLMs show that 8-bit or lower weight quantization is feasible with minimal impact on task performance with full-precision activation. Nevertheless, weight-only quantization does not leverage the potential acceleration on general edge processors, which typically support 16x16 and 8x8 integer multipliers. Additionally, it cannot benefit from specialized devices like Field-Programmable Gate Arrays (FPGAs), which offer reconfigure computing to design custom multipliers of arbitrary bit width. In this paper, we propose HotaQ, a Hardware-oriented token adaptive Quantization framework for general LLMs, and implement end-to-end accelerators on multiple devices. Considering the hardware profiling and activation analysis, we first introduce a basic activation quantization strategy to balance the trade-off of task performance and real inference speed. Then we integrate the activation-aware token pruning workflow to reduce the outliers and the adverse impact on attentivity. For mobile devices, we notably design the SIMD-based 4-bit multiplier and the efficient TRIP matrix multiplication for mobile devices. For FPGAs, we specially utilize the DSP packing techniques for 4/8-bit systolic-array-based multipliers and introduce an analytical model for achieving the best performance with limited on-chip resources. We apply our framework on different scales of LLMs, including LLaMA, OPT, and BLOOM with 4/8-bit for the activation and 4-bit for the weight. Experiments show that HotaQ achieves quantization on both model weights and activations while maintaining task performance comparable to existing weight-only quantization methods or even FP16 models. In the 4/8-bit scenario, HotaQ achieves an on-device speedup of up to 2.5× and 5.2× compared to its FP16 counterparts across edge devices and FPGAs respectively, marking a pioneering advancement in this domain.

ASLog: an Area-Efficient CNN Accelerator for Per-Channel Logarithmic Post-Training Quantization

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

A Reconfigurable Approximate Multiplier for Quantized CNN Applications.

HotaQ: Hardware Oriented Token Adaptive Quantization for Large Language Models

A Memory-Efficient CNN Accelerator Using Segmented Logarithmic Quantization and Multi-Cluster Architecture

Algorithm-Hardware Co-Design of Distribution-Aware Logarithmic-Posit Encodings for Efficient DNN Inference

A hardware-friendly logarithmic quantization method for CNNs and FPGA implementation

An Efficient Streaming Accelerator for Low Bit-Width Convolutional Neural Networks

Base-Reconfigurable Segmented Logarithmic Quantization and Hardware Design for Deep Neural Networks

AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer

A Low-Power Sparse Convolutional Neural Network Accelerator with Pre-Encoding Radix-4 Booth Multiplier

A Low-Power Arithmetic Element for Multi-Base Logarithmic Computation on Deep Neural Networks

Adaptive Global Power-of-Two Ternary Quantization Algorithm Based on Unfixed Boundary Thresholds

GroupQ: Group-Wise Quantization With Multi-Objective Optimization for CNN Accelerators

Custom Network Quantization Method for Lightweight CNN Acceleration on FPGAs

A Logarithmic Floating-Point Multiplier for the Efficient Training of Neural Networks

A Hardware-Friendly Low-Bit Power-of-Two Quantization Method for CNNs and Its FPGA Implementation

Power Efficient Tiny Yolo CNN Using Reduced Hardware Resources Based on Booth Multiplier and WALLACE Tree Adders

High PE Utilization CNN Accelerator with Channel Fusion Supporting Pattern-Compressed Sparse Neural Networks

PQA: Exploring the Potential of Product Quantization in DNN Hardware Acceleration