Abstract:Advances in deep learning have led to state-of-the-art performance across a multitude of speech recognition tasks. Nevertheless, the widespread deployment of deep neural networks for on-device speech recognition remains a challenge, particularly in edge scenarios where the memory and computing resources are highly constrained (e.g., low-power embedded devices) or where the memory and computing budget dedicated to speech recognition is low (e.g., mobile devices performing numerous tasks besides speech recognition). In this study, we introduce the concept of attention condensers for building low-footprint, highly-efficient deep neural networks for on-device speech recognition on the edge. An attention condenser is a self-attention mechanism that learns and produces a condensed embedding characterizing joint local and cross-channel activation relationships, and performs selective attention accordingly. To illustrate its efficacy, we introduce TinySpeech, low-precision deep neural networks comprising largely of attention condensers tailored for on-device speech recognition using a machine-driven design exploration strategy, with one tailored specifically with microcontroller operation constraints. Experimental results on the Google Speech Commands benchmark dataset for limited-vocabulary speech recognition showed that TinySpeech networks achieved significantly lower architectural complexity (as much as $507\times$ fewer parameters), lower computational complexity (as much as $48\times$ fewer multiply-add operations), and lower storage requirements (as much as $2028\times$ lower weight memory requirements) when compared to previous work. These results not only demonstrate the efficacy of attention condensers for building highly efficient networks for on-device speech recognition, but also illuminate its potential for accelerating deep learning on the edge and empowering TinyML applications.

A Fixed-Point Neural Network Architecture for Speech Applications on Resource Constrained Hardware

A fixed-point neural network for keyword detection on resource constrained hardware

A Low-Power In-Memory Multiplication and Accumulation Array with Modified Radix-4 Input and Canonical Signed Digit Weights

Low-power Neuromorphic Speech Recognition Engine with Coarse-Grain Sparsity.

More is Less: Domain-Specific Speech Recognition Microprocessor Using One-Dimensional Convolutional Recurrent Neural Network

An Ultra-Low Power Binarized Convolutional Neural Network-Based Speech Recognition Processor with On-Chip Self-Learning.

Hello Edge: Keyword Spotting on Microcontrollers

9.1 μW keyword spotting processor based on optimized MFCC and small‐footprint TENet in 28‐nm CMOS

Efficient Binary Weight Convolutional Network Accelerator for Speech Recognition

Optimizing Speech Recognition For The Edge

Residual Spiking Neural Network on a Programmable Neuromorphic Hardware for Speech Keyword Spotting

A Fully Integrated 1.7mw Attention-Based Automatic Speech Recognition Processor

TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices

Digit-Serial DA-Based Fixed-Point RNNs: A Unified Approach for Enhancing Architectural Efficiency

A 5.1pJ/Neuron 127.3us/Inference RNN-based Speech Recognition Processor using 16 Computing-in-Memory SRAM Macros in 65nm CMOS

A Hybrid Precision Low Power Computing-in-memory Architecture for Neural Networks

Binary Neural Networks for Speech Recognition

Puncturing the Memory Wall

NS-FDN: Near-Sensor Processing Architecture of Feature-Configurable Distributed Network for Beyond-Real-Time Always-on Keyword Spotting

Model compression applied to small-footprint keyword spotting

Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models