Abstract:In this paper, a multilingual end-to-end framework, called ATCSpeechNet, is proposed to tackle the issue of translating communication speech into human-readable text in air traffic control (ATC) systems. In the proposed framework, we focus on integrating multilingual automatic speech recognition (ASR) into one model, in which an end-to-end paradigm is developed to convert speech waveforms into text directly, without any feature engineering or lexicon. To compensate the deficiency of handcrafted feature engineering caused by ATC challenges, including multilingual, multispeaker dialog and unstable speech rates, a speech representation learning (SRL) network is proposed to capture robust and discriminative speech representations from raw waves. The self-supervised training strategy is adopted to optimize the SRL network from unlabeled data, and to further predict the speech features, i.e., wave-to-feature. An end-to-end architecture is improved to complete the ASR task, in which a grapheme-based modeling unit is applied to address the multilingual ASR issue. Facing the problem of small transcribed samples in the ATC domain, an unsupervised approach with mask prediction is applied to pretrain the backbone network of the ASR model on unlabeled data by a feature-to-feature process. Finally, by integrating the SRL with ASR, an end-to-end multilingual ASR framework is formulated in a supervised manner, which is able to translate the raw wave into text in one model, i.e., wave-to-text. Experimental results on the ATCSpeech corpus demonstrate that the proposed approach achieves high performance with a very small labeled corpus and less resource consumption, only a 4.20% label error rate on the 58-hour transcribed corpus. Compared to the baseline model, the proposed approach obtains over 100% relative performance improvement which can be further enhanced with increasing size of the transcribed samples. It is also confirmed that the proposed SRL and training strategies make significant contributions to improving the final performance. In addition, the effectiveness of the proposed framework is also validated on common corpora (AISHELL, LibriSpeech, and cv-fr). More importantly, the proposed multilingual framework not only reduces the system complexity but also obtains higher accuracy compared to that of the independent monolingual ASR models. The proposed approach can also greatly reduce the cost of annotating samples, which benefits to advance the ASR technique to industrial applications.

ExKaldi-RT: A Real-Time Automatic Speech Recognition Extension Toolkit of Kaldi

ExKaldi - A Python-based Extension Tool of Kaldi.

The Kaldi Speech Recognition Toolkit

PyKaldi2: Yet another speech toolkit based on Kaldi and PyTorch

Espresso: A Fast End-to-end Neural Speech Recognition Toolkit.

CAT: A CTC-CRF Based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches Towards Data Efficiency and Low Latency

Real-time Architecture for Audio-Visual Active Speaker Detection.

CAT: CRF-based ASR Toolkit

KoSpeech: Open-Source Toolkit for End-to-End Korean Speech Recognition

SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale

Kaldi+PDNN: Building DNN-based ASR Systems with Kaldi and PDNN

Using Kaldi for Automatic Speech Recognition of Conversational Austrian German

SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation

AV Taris: Online Audio-Visual Speech Recognition

Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data

Real-time End-to-End Monaural Multi-speaker Speech Recognition

RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognition

FunASR: A Fundamental End-to-End Speech Recognition Toolkit

ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation

ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems

Attention-based Transducer for Online Speech Recognition