Abstract:In this paper, a multilingual end-to-end framework, called ATCSpeechNet, is proposed to tackle the issue of translating communication speech into human-readable text in air traffic control (ATC) systems. In the proposed framework, we focus on integrating multilingual automatic speech recognition (ASR) into one model, in which an end-to-end paradigm is developed to convert speech waveforms into text directly, without any feature engineering or lexicon. To compensate the deficiency of handcrafted feature engineering caused by ATC challenges, including multilingual, multispeaker dialog and unstable speech rates, a speech representation learning (SRL) network is proposed to capture robust and discriminative speech representations from raw waves. The self-supervised training strategy is adopted to optimize the SRL network from unlabeled data, and to further predict the speech features, i.e., wave-to-feature. An end-to-end architecture is improved to complete the ASR task, in which a grapheme-based modeling unit is applied to address the multilingual ASR issue. Facing the problem of small transcribed samples in the ATC domain, an unsupervised approach with mask prediction is applied to pretrain the backbone network of the ASR model on unlabeled data by a feature-to-feature process. Finally, by integrating the SRL with ASR, an end-to-end multilingual ASR framework is formulated in a supervised manner, which is able to translate the raw wave into text in one model, i.e., wave-to-text. Experimental results on the ATCSpeech corpus demonstrate that the proposed approach achieves high performance with a very small labeled corpus and less resource consumption, only a 4.20% label error rate on the 58-hour transcribed corpus. Compared to the baseline model, the proposed approach obtains over 100% relative performance improvement which can be further enhanced with increasing size of the transcribed samples. It is also confirmed that the proposed SRL and training strategies make significant contributions to improving the final performance. In addition, the effectiveness of the proposed framework is also validated on common corpora (AISHELL, LibriSpeech, and cv-fr). More importantly, the proposed multilingual framework not only reduces the system complexity but also obtains higher accuracy compared to that of the independent monolingual ASR models. The proposed approach can also greatly reduce the cost of annotating samples, which benefits to advance the ASR technique to industrial applications.

LIS-Net: An end-to-end light interior search network for speech command recognition

SAR-Net: A End-to-End Deep Speech Accent Recognition Network

CACnet: Cube Attentional CNN for Automatic Speech Recognition

DLD: An Optimized Chinese Speech Recognition Model Based on Deep Learning

Exploiting Hybrid Models of Tensor-Train Networks for Spoken Command Recognition

Towards Efficient Recurrent Architectures: A Deep LSTM Neural Network Applied to Speech Enhancement and Recognition

Deep causal speech enhancement and recognition using efficient long-short term memory Recurrent Neural Network

Deep Residual-Dense Lattice Network for Speech Enhancement

Multi-Channel Automatic Speech Recognition Using Deep Complex Unet

End-to-End Speech Recognition Model Based on Dilated Sparse Aware Network

ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems

CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding

Lightspeech: Lightweight and Fast Text to Speech with Neural Architecture Search

Extremely Low Footprint End-to-End ASR System for Smart Device

LiSenNet: Lightweight Sub-band and Dual-Path Modeling for Real-Time Speech Enhancement

Espresso: A Fast End-to-end Neural Speech Recognition Toolkit.

LCSNet: End-to-End Lipreading with Channel-aware Feature Selection

Voice Interaction Recognition Design in Real-Life Scenario Mobile Robot Applications

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin