Abstract:In this paper, a multilingual end-to-end framework, called ATCSpeechNet, is proposed to tackle the issue of translating communication speech into human-readable text in air traffic control (ATC) systems. In the proposed framework, we focus on integrating multilingual automatic speech recognition (ASR) into one model, in which an end-to-end paradigm is developed to convert speech waveforms into text directly, without any feature engineering or lexicon. To compensate the deficiency of handcrafted feature engineering caused by ATC challenges, including multilingual, multispeaker dialog and unstable speech rates, a speech representation learning (SRL) network is proposed to capture robust and discriminative speech representations from raw waves. The self-supervised training strategy is adopted to optimize the SRL network from unlabeled data, and to further predict the speech features, i.e., wave-to-feature. An end-to-end architecture is improved to complete the ASR task, in which a grapheme-based modeling unit is applied to address the multilingual ASR issue. Facing the problem of small transcribed samples in the ATC domain, an unsupervised approach with mask prediction is applied to pretrain the backbone network of the ASR model on unlabeled data by a feature-to-feature process. Finally, by integrating the SRL with ASR, an end-to-end multilingual ASR framework is formulated in a supervised manner, which is able to translate the raw wave into text in one model, i.e., wave-to-text. Experimental results on the ATCSpeech corpus demonstrate that the proposed approach achieves high performance with a very small labeled corpus and less resource consumption, only a 4.20% label error rate on the 58-hour transcribed corpus. Compared to the baseline model, the proposed approach obtains over 100% relative performance improvement which can be further enhanced with increasing size of the transcribed samples. It is also confirmed that the proposed SRL and training strategies make significant contributions to improving the final performance. In addition, the effectiveness of the proposed framework is also validated on common corpora (AISHELL, LibriSpeech, and cv-fr). More importantly, the proposed multilingual framework not only reduces the system complexity but also obtains higher accuracy compared to that of the independent monolingual ASR models. The proposed approach can also greatly reduce the cost of annotating samples, which benefits to advance the ASR technique to industrial applications.

Integrating Source-Channel and Attention-Based Sequence-to-Sequence Models for Speech Recognition

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis

Automatic channel selection and spatial feature integration for multi-channel speech recognition across various array topologies

Speaker conditioned acoustic modeling for multi-speaker conversational ASR

State-of-the-Art Speech Recognition with Sequence-to-Sequence Models

Combining Frame-Synchronous and Label-Synchronous Systems for Speech Recognition

CASA-ASR: Context-Aware Speaker-Attributed ASR.

A Comparative Study on Multichannel Speaker-Attributed Automatic Speech Recognition in Multi-party Meetings

Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition

Multi-Frame Cross-Channel Attention and Speaker Diarization Based Speaker-Attributed Automatic Speech Recognition System for Multi-Channel Multi-Party Meeting Transcription

Exploring Attention Mechanisms Based on Summary Information for End-to-end Automatic Speech Recognition

End-to-End Multi-speaker ASR with Independent Vector Analysis

Multi-encoder multi-resolution framework for end-to-end speech recognition

Combining Hybrid DNN-HMM ASR Systems with Attention-Based Models Using Lattice Rescoring

Multichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition

Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond

A Unified Recognition and Correction Model under Noisy and Accent Speech Conditions

AISPEECH-SJTU ASR SYSTEM FOR THE ACCENTED ENGLISH SPEECH RECOGNITION CHALLENGE

ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems

MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition