Abstract:In this paper, a multilingual end-to-end framework, called ATCSpeechNet, is proposed to tackle the issue of translating communication speech into human-readable text in air traffic control (ATC) systems. In the proposed framework, we focus on integrating multilingual automatic speech recognition (ASR) into one model, in which an end-to-end paradigm is developed to convert speech waveforms into text directly, without any feature engineering or lexicon. To compensate the deficiency of handcrafted feature engineering caused by ATC challenges, including multilingual, multispeaker dialog and unstable speech rates, a speech representation learning (SRL) network is proposed to capture robust and discriminative speech representations from raw waves. The self-supervised training strategy is adopted to optimize the SRL network from unlabeled data, and to further predict the speech features, i.e., wave-to-feature. An end-to-end architecture is improved to complete the ASR task, in which a grapheme-based modeling unit is applied to address the multilingual ASR issue. Facing the problem of small transcribed samples in the ATC domain, an unsupervised approach with mask prediction is applied to pretrain the backbone network of the ASR model on unlabeled data by a feature-to-feature process. Finally, by integrating the SRL with ASR, an end-to-end multilingual ASR framework is formulated in a supervised manner, which is able to translate the raw wave into text in one model, i.e., wave-to-text. Experimental results on the ATCSpeech corpus demonstrate that the proposed approach achieves high performance with a very small labeled corpus and less resource consumption, only a 4.20% label error rate on the 58-hour transcribed corpus. Compared to the baseline model, the proposed approach obtains over 100% relative performance improvement which can be further enhanced with increasing size of the transcribed samples. It is also confirmed that the proposed SRL and training strategies make significant contributions to improving the final performance. In addition, the effectiveness of the proposed framework is also validated on common corpora (AISHELL, LibriSpeech, and cv-fr). More importantly, the proposed multilingual framework not only reduces the system complexity but also obtains higher accuracy compared to that of the independent monolingual ASR models. The proposed approach can also greatly reduce the cost of annotating samples, which benefits to advance the ASR technique to industrial applications.

A Real-Time ATC Safety Monitoring Framework Using a Deep Learning Approach

Data-driven Traffic Dynamic Understanding and Safety Monitoring Applications

A Unified Framework for Multilingual Speech Recognition in Air Traffic Control Systems

A Deep Learning Method for Lane Changing Situation Assessment and Decision Making.

An Aircraft Collision Avoidance Method Based on Deep Reinforcement Learning

Enhancing Air Traffic Control: A Transparent Deep Reinforcement Learning Framework for Autonomous Conflict Resolution

Speech Recognition for Air Traffic Control Via Feature Learning and End-to-end Training

A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers

Enhancing Air Traffic Control Communication Systems with Integrated Automatic Speech Recognition: Models, Applications and Performance Evaluation

An integrated framework for eye tracking-assisted task capability recognition of air traffic controllers with machine learning

ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems

Identifying and managing risks of AI-driven operations: A case study of automatic speech recognition for improving air traffic safety

Deep Reinforcement Learning Based Conflict Detection and Resolution in Air Traffic Control

Deep4Air: A Novel Deep Learning Framework for Airport Airside Surveillance

A Deep Ensemble Multi-Agent Reinforcement Learning Approach for Air Traffic Control

A BERT-based Intent Recognition and Slot Filling Joint Model for Air Traffic Control Instruction Understanding

Autonomous air traffic separation assurance through machine learning

Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic control

Improving speech recognition models with small samples for air traffic control systems

Autonomous Air Traffic Controller: A Deep Multi-Agent Reinforcement Learning Approach

Air Traffic Control Command Monitoring System Based on Information Integration