Abstract:As one of the prevailing deep neural networks compression techniques, layer-wise mixed-precision quantization (MPQ) strikes a better balance between accuracy and efficiency than uniform quantization schemes. However, existing MPQ strategies either lack hardware awareness or incur huge computation costs, limiting their deployment at the edge. Additionally, researchers usually make a one-time decision between post-training quantization (PTQ) and quantization-aware training (QAT) based on the quantized bit-width or hardware requirements. In this paper, we propose the tight integration of versatile MPQ inference units supporting INT2-INT8 and INT16 precisions, which feature a hierarchical multiplier architecture, into a RISC-V processor pipeline through micro-architecture and Instruction Set Architecture (ISA) co-design. Synthesized with a 14nm technology, the design delivers a speedup of 15.50× to 47.67× over the baseline RV64IMA core when running a single convolution layer kernel and achieves up to 2.86 GOPS performance. This work also achieves an energy efficiency at 20.51 TOPS/W, which not only exceeds contemporary state-of-the-art MPQ hardware solutions at the edge, but also marks a significant advancement in the field. We also propose a novel MPQ search algorithm that incorporates both hardware awareness and training necessity. The algorithm samples layer-wise sensitivities using a set of newly proposed metrics and runs a heuristics search. Evaluation results show that this search algorithm achieves 2.2%∼6.7% higher inference accuracy under similar hardware constraints compared to state-of-the-art MPQ strategies. Furthermore we expand the search space using a dynamic programming (DP) strategy to perform search with more fine-grained accuracy intervals and support multi-dimensional search. This further improves the inference accuracy by over 1.3% compared to a greedy-based search.

Communication-aware Quantization for Deep Learning Inference Parallelization on Chiplet-based Accelerators

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

Pse: Mixed Quantization Framework of Neural Networks for Efficient Deployment

Single-shot Pruning and Quantization for Hardware-Friendly Neural Network Acceleration

CAP: Communication-aware Automated Parallelization for Deep Learning Inference on CMP Architectures

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

PalQuant: Accelerating High-Precision Networks on Low-Precision Accelerators

On-Chip Hardware-Aware Quantization for Mixed Precision Neural Networks

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Cloud-Edge Inference under Communication Constraints: Data Quantization and Early Exit.

Hardware-Centric AutoML for Mixed-Precision Quantization

Exploring Quantization and Mapping Synergy in Hardware-Aware Deep Neural Network Accelerators

A Fine-Grained End-to-End Latency Optimization Framework for Wireless Collaborative Inference

Edge-MPQ: Layer-Wise Mixed-Precision Quantization With Tightly Integrated Versatile Inference Units for Edge Computing

DQI: A Dynamic Quantization Method for Efficient Convolutional Neural Network Inference Accelerators

QUIDAM: A Framework for Quantization-aware DNN Accelerator and Model Co-Exploration

Mixed Precision Quantized Neural Network Accelerator for Remote Sensing Images Classification

"It is okay to be uncommon": Quantizing Sound Event Detection Networks on Hardware Accelerators with Uncommon Sub-Byte Support

An Efficient Streaming Accelerator for Low Bit-Width Convolutional Neural Networks

GroupQ: Group-Wise Quantization With Multi-Objective Optimization for CNN Accelerators

Accelerating Neural Network Inference by Overflow Aware Quantization