Abstract:Transformer networks are rapidly becoming SotA in many fields, such as NLP and CV. Similarly to CNN, there is a strong push for deploying Transformer models at the extreme edge, ultimately fitting the tiny power budget and memory footprint of MCUs. However, the early approaches in this direction are mostly ad-hoc, platform, and model-specific. This work aims to enable and optimize the flexible, multi-platform deployment of encoder Tiny Transformers on commercial MCUs. We propose a complete framework to perform end-to-end deployment of Transformer models onto single and multi-core MCUs. Our framework provides an optimized library of kernels to maximize data reuse and avoid unnecessary data marshaling operations into the crucial attention block. A novel MHSA inference schedule, named Fused-Weight Self-Attention, is introduced, fusing the linear projection weights offline to further reduce the number of operations and parameters. Furthermore, to mitigate the memory peak reached by the computation of the attention map, we present a Depth-First Tiling scheme for MHSA. We evaluate our framework on three different MCU classes exploiting ARM and RISC-V ISA, namely the STM32H7, the STM32L4, and GAP9 (RV32IMC-XpulpV2). We reach an average of 4.79x and 2.0x lower latency compared to SotA libraries CMSIS-NN (ARM) and PULP-NN (RISC-V), respectively. Moreover, we show that our MHSA depth-first tiling scheme reduces the memory peak by up to 6.19x, while the fused-weight attention can reduce the runtime by 1.53x, and number of parameters by 25%. We report significant improvements across several Tiny Transformers: for instance, when executing a transformer block for the task of radar-based hand-gesture recognition on GAP9, we achieve a latency of 0.14ms and energy consumption of 4.92 micro-joules, 2.32x lower than the SotA PULP-NN library on the same platform.

MLonMCU: TinyML Benchmarking with Fast Retargeting

TinyML Platforms Benchmarking

Toward Attention-based TinyML: A Heterogeneous Accelerated Architecture and Automated Deployment Flow

MATCH: Model-Aware TVM-based Compilation for Heterogeneous Edge Devices

ML-MCU: A Framework to Train ML Classifiers on MCU-based IoT Edge Devices

iMCU: A 28-nm Digital In-Memory Computing-Based Microcontroller Unit for TinyML

Low-Energy On-Device Personalization for MCUs

Automated deep‐learning model optimization framework for microcontrollers

Optimizing the Deployment of Tiny Transformers on Low-Power MCUs

Accelerating TinyML Inference on Microcontrollers through Approximate Kernels

Decoupled Access-Execute enabled DVFS for tinyML deployments on STM32 microcontrollers

TinyOL: TinyML with Online-Learning on Microcontrollers

Deeploy: Enabling Energy-Efficient Deployment of Small Language Models On Heterogeneous Microcontrollers

Tiny Machine Learning: Progress and Futures

TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems.

TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices

A Continual and Incremental Learning Approach for TinyML On-device Training Using Dataset Distillation and Model Size Adaption

MCUBench: A Benchmark of Tiny Object Detectors on MCUs

Custom Hardware Inference Accelerator for TensorFlow Lite for Microcontrollers

RIOT-ML: toolkit for over-the-air secure updates and performance evaluation of TinyML models

Robustifying the Deployment of tinyML Models for Autonomous Mini-Vehicles