Abstract:This study explores the feasibility of constructing a small-scale speech recognition system capable of competing with larger, modern automated speech recognition (ASR) systems in both performance and word error rate (WER). Our central hypothesis posits that a compact transformer-based ASR model can yield comparable results, specifically in terms of WER, to traditional ASR models while challenging contemporary ASR systems that boast significantly larger computational sizes. The aim is to extend ASR capabilities to under-resourced languages with limited corpora, catering to scenarios where practitioners face constraints in both data availability and computational resources. The model, comprising a compact convolutional neural network (CNN) and transformer architecture with 2.214 million parameters, challenges the conventional wisdom that large-scale transformer-based ASR systems are essential for achieving high accuracy. In comparison, contemporary ASR systems often deploy over 300 million parameters. Trained on a modest dataset of approximately 3000 h – significantly less than the 50,000 h used in larger systems – the proposed model leverages the Common Voice and LibriSpeech datasets. Evaluation on the LibriSpeech test-clean and test-other datasets produced character error rates (CERs) of 6.40% and 16.73% and WERs of 16.03% and 35.51% respectively. Comparisons with existing architectures showcase the efficiency of our model. A gated recurrent unit (GRU) architecture, albeit achieving lower error rates, incurred a computational cost 24 times larger than our proposed model. Large-scale transformer architectures, while achieving marginally lower WERs (2%–4% on LibriSpeech test-clean), require 200 times more parameters and 53,000 additional hours of training data. Modern large language models are used to improve the WERs, but require large computational resources. To further enhance performance, a small 4-g language model was integrated into our end-to-end ASR model, resulting in improved WERs. The overarching goal of this work is to provide a practical solution for practitioners dealing with limited datasets and computational resources, particularly in the context of under-resourced languages.

Non-autoregressive Transformer-based End-to-end ASR using BERT

Improving Non-Autoregressive End-to-End Speech Recognition with Pre-Trained Acoustic and Language Models

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

A Lexical-aware Non-autoregressive Transformer-based ASR Model

Innovative Bert-based Reranking Language Models for Speech Recognition

A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition

End-to-End Speech Recognition with Pre-trained Masked Language Model

Beyond Universal Transformer: block reusing with adaptor in Transformer for automatic speech recognition

Hybrid Autoregressive and Non-Autoregressive Transformer Models for Speech Recognition

End-to-End ASR with Adaptive Span Self-Attention

End-to-end automated speech recognition using a character based small scale transformer architecture

TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding

NON-AUTOREGRESSIVE TRANSFORMER WITH UNIFIED BIDIRECTIONAL DECODER FOR AUTOMATIC SPEECH RECOGNITION

Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism

Transformer with Bidirectional Decoder for Speech Recognition

Effective Cross-Utterance Language Modeling for Conversational Speech Recognition

Dual-Branch Attention-In-Attention Transformer for Single-Channel Speech Enhancement

Transformer-based Model for ASR N-Best Rescoring and Rewriting

Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search

Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition

A Transformer-Based End-to-End Automatic Speech Recognition Algorithm