Abstract:Accents, as variations from standard pronunciation, pose significant challenges for speech recognition systems. Although joint automatic speech recognition (ASR) and accent recognition (AR) training has been proven effective in handling multi-accent scenarios, current multi-task ASR-AR approaches overlook the granularity differences between tasks. Fine-grained units capture pronunciation-related accent characteristics, while coarse-grained units are better for learning linguistic information. Moreover, an explicit interaction of two tasks can also provide complementary information and improve the performance of each other, but it is rarely used by existing approaches. In this paper, we propose a novel Decoupling and Interacting Multi-task Network (DIMNet) for joint speech and accent recognition, which is comprised of a connectionist temporal classification (CTC) branch, an AR branch, an ASR branch, and a bottom feature encoder. Specifically, AR and ASR are first decoupled by separated branches and two-granular modeling units to learn task-specific representations. The AR branch is from our previously proposed linguistic-acoustic bimodal AR model and the ASR branch is an encoder-decoder based Conformer model. Then, for the task interaction, the CTC branch provides aligned text for the AR task, while accent embeddings extracted from our AR model are incorporated into the ASR branch's encoder and decoder. Finally, during ASR inference, a cross-granular rescoring method is introduced to fuse the complementary information from the CTC and attention decoder after the decoupling. Our experiments on English and Chinese datasets demonstrate the effectiveness of the proposed model, which achieves 21.45%/28.53% AR accuracy relative improvement and 32.33%/14.55% ASR error rate relative reduction over a published standard baseline, respectively.

Multi-task Recurrent Model for True Multilingual Speech Recognition

Cross-Lingual Text Image Recognition Via Multi-Task Sequence to Sequence Learning.

Towards Language-Universal Mandarin-English Speech Recognition

Collaborative Learning for Language and Speaker Recognition.

Improved BLSTM RNN Based Accent Speech Recognition Using Multi-task Learning and Accent Embeddings

Collaborative Joint Training With Multitask Recurrent Model for Speech and Speaker Recognition.

Mandarin-English bilingual phone modeling and combining MPE based Discriminative training for cross-language speech recognition

Multi-Task Transformer with Adaptive Cross-Entropy Loss for Multi-Dialect Speech Recognition

Improving Accented Mandarin Speech Recognition by Using Recurrent Neural Network Based Language Model Adaptation

Phone modeling and combining discriminative training for Mandarin-English bilingual speech recognition

Adapting Multi-Lingual ASR Models for Handling Multiple Talkers

Recurrent Neural Network Based Language Model Adaptation for Accent Mandarin Speech.

Real-time End-to-End Monaural Multi-speaker Speech Recognition

Multi-Task Learning for Multiple Language Translation.

Multilingual Approach to Joint Speech and Accent Recognition with DNN-HMM Framework

Decoupling and Interacting Multi-Task Learning Network for Joint Speech and Accent Recognition

Enhancing CTC-based speech recognition with diverse modeling units

Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition

A hybrid CTC+Attention model based on end-to-end framework for multilingual speech recognition

Tri-stage training with language-specific encoder and bilingual acoustic learner for code-switching speech recognition

FC-MTLF: A Fine- and Coarse-grained Multi-Task Learning Framework for Cross-Lingual Spoken Language Understanding.