Abstract:In recent years, the need for the efficient deployment of Neural Networks (NN) on edge devices has been steadily increasing. However, the high computational demand required for Machine Learning (ML) inference on tiny microcontroller-based IoT devices avoids a direct software deployment on such resource-constrained edge devices. Therefore, various custom and application-specific NN hardware accelerators have been proposed to enable real-time Machine Learning (ML) inference on low-power and resource-limited edge devices. Efficient mapping of the computational load onto hardware and software resources is a key challenge for performance improvement while keeping low power and a low area footprint. High performance and yet low power embedded processors may be attained via the usage of hardware acceleration. This paper presents an efficient hardware-software framework to accelerate machine learning inference on edge devices using a modified TensorFlow Lite for Microcontroller (TFLM) model running on a Microcontroller (MCU) and a dedicated Neural Processing Unit (NPU) custom hardware accelerator, referred to as MCU-NPU. The proposed framework supports weight compression of pruned quantized NN models and exploits the pruned model sparsity to reduce computational complexity further. The proposed methodology has been evaluated by employing the MCU-NPU acceleration for various TFLM-based NN architectures using the common MLPerf Tiny benchmark. Experimental results demonstrate a significant speedup of up to 724x compared to a pure software implementation. For example, the resulting runtime for the CIFAR-10 classification is reduced from about 20 sec to only 37 ms using the proposed hardware acceleration. Moreover, the proposed hardware accelerator outperforms all the reference models optimized for edge devices in terms of inference runtime.

Efficient Neural Network Deployment for Microcontroller

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Deep Compression for PyTorch Model Deployment on Microcontrollers

Neural networks on microcontrollers: saving memory at inference via operator reordering

Automated deep‐learning model optimization framework for microcontrollers

Differentiable Network Pruning for Microcontrollers

Energy-efficient Deployment of Deep Learning Applications on Cortex-M based Microcontrollers using Deep Compression

Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review

SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers

Decoupled Access-Execute enabled DVFS for tinyML deployments on STM32 microcontrollers

Enabling Efficient Deep Learning on MCU with Transient Redundancy Elimination

MicroENet: an Efficient Network for MCUs with Low Model Parameters and Peak Memory

Enabling High Performance Deep Learning Networks on Embedded Systems

MCUFormer: Deploying Vision Tranformers on Microcontrollers with Limited Memory.

MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory

vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs

Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms

Evaluation of Convolution Primitives for Embedded Neural Networks on 32-bit Microcontrollers

Custom Hardware Inference Accelerator for TensorFlow Lite for Microcontrollers

Enabling Large Neural Networks on Tiny Microcontrollers with Swapping

MCUNet: Tiny Deep Learning on IoT Devices