Abstract:Accents, as variations from standard pronunciation, pose significant challenges for speech recognition systems. Although joint automatic speech recognition (ASR) and accent recognition (AR) training has been proven effective in handling multi-accent scenarios, current multi-task ASR-AR approaches overlook the granularity differences between tasks. Fine-grained units capture pronunciation-related accent characteristics, while coarse-grained units are better for learning linguistic information. Moreover, an explicit interaction of two tasks can also provide complementary information and improve the performance of each other, but it is rarely used by existing approaches. In this paper, we propose a novel Decoupling and Interacting Multi-task Network (DIMNet) for joint speech and accent recognition, which is comprised of a connectionist temporal classification (CTC) branch, an AR branch, an ASR branch, and a bottom feature encoder. Specifically, AR and ASR are first decoupled by separated branches and two-granular modeling units to learn task-specific representations. The AR branch is from our previously proposed linguistic-acoustic bimodal AR model and the ASR branch is an encoder-decoder based Conformer model. Then, for the task interaction, the CTC branch provides aligned text for the AR task, while accent embeddings extracted from our AR model are incorporated into the ASR branch's encoder and decoder. Finally, during ASR inference, a cross-granular rescoring method is introduced to fuse the complementary information from the CTC and attention decoder after the decoupling. Our experiments on English and Chinese datasets demonstrate the effectiveness of the proposed model, which achieves 21.45%/28.53% AR accuracy relative improvement and 32.33%/14.55% ASR error rate relative reduction over a published standard baseline, respectively.

Collaborative Joint Training With Multitask Recurrent Model for Speech and Speaker Recognition.

Collaborative Learning for Language and Speaker Recognition.

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Multi-task Recurrent Model for True Multilingual Speech Recognition

A Speaker-Dependent Approach to Single-Channel Joint Speech Separation and Acoustic Modeling Based on Deep Neural Networks for Robust Recognition of Multi-Talker Speech

Joint Speech-Text Embeddings for Multitask Speech Processing

A Speaker-Dependent Deep Learning Approach to Joint Speech Separation and Acoustic Modeling for Multi-Talker Automatic Speech Recognition

Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcription

Neural Network Based Multi-Factor Aware Joint Training for Robust Speech Recognition.

An efficient joint training model for monaural noisy-reverberant speech recognition

Decoupling and Interacting Multi-Task Learning Network for Joint Speech and Accent Recognition

Multi-task Joint-Learning of Deep Neural Networks for Robust Speech Recognition

Progressive Joint Modeling in Unsupervised Single-Channel Overlapped Speech Recognition.

Audio Visual Speech Recognition with Multimodal Recurrent Neural Networks

SADDEL: Joint Speech Separation and Denoising Model based on Multitask Learning

Collaborative Training of Acoustic Encoders for Speech Recognition

Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition

Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognition

A Scene-Associated Training Method for Mobile Robot Speech Recognition in Multisource Reverberated Environments.

Multimodal Pre-Train then Transfer Learning Approach for Speaker Recognition