Abstract:Deep neural speech and audio processing systems have a large number of trainable parameters, a relatively complex architecture, and require a vast amount of training data and computational power. These constraints make it more challenging to integrate such systems into embedded devices and utilize them for real-time, real-world applications. We tackle these limitations by introducing DeepSpectrumLite, an open-source, lightweight transfer learning framework for on-device speech and audio recognition using pre-trained image Convolutional Neural Networks (CNNs). The framework creates and augments Mel spectrogram plots on the fly from raw audio signals which are then used to finetune specific pre-trained CNNs for the target classification task. Subsequently, the whole pipeline can be run in real-time with a mean inference lag of 242.0 ms when a DenseNet121 model is used on a consumer-grade Motorola moto e7 plus smartphone. DeepSpectrumLite operates decentralized, eliminating the need for data upload for further processing. We demonstrate the suitability of the proposed transfer learning approach for embedded audio signal processing by obtaining state-of-the-art results on a set of paralinguistic and general audio tasks, including speech and music emotion recognition, social signal processing, COVID-19 cough and COVID-19 speech analysis, and snore sound classification. We provide an extensive command-line interface for users and developers which is comprehensively documented and publicly available at https://github.com/DeepSpectrum/DeepSpectrumLite.

Espresso: A Fast End-to-end Neural Speech Recognition Toolkit.

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding

ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models

SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale

FunASR: A Fundamental End-to-End Speech Recognition Toolkit

End-to-End Architectures for Speech Recognition

Extremely Low Footprint End-to-End ASR System for Smart Device

Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model

EURO: ESPnet Unsupervised ASR Open-source Toolkit

Deep Speaker: an End-to-End Neural Speaker Embedding System

Non-Autoregressive End-To-End Automatic Speech Recognition Incorporating Downstream Natural Language Processing

End-to-End Deep Learning Speech Recognition Model for Silent Speech Challenge

Fast End-to-End Speech Recognition Via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT

FastSpeech: Fast, Robust and Controllable Text to Speech

DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing From Decentralized Data

Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems

End-To-End deep neural models for Automatic Speech Recognition for Polish Language

KoSpeech: Open-Source Toolkit for End-to-End Korean Speech Recognition

SpeechBrain: A General-Purpose Speech Toolkit

LIS-Net: An end-to-end light interior search network for speech command recognition