Abstract:The variety and complexity of accents pose a huge challenge to robust Automatic Speech Recognition (ASR). Some previous work has attempted to address such problems, however most of the current approaches either require prior knowledge about the target accent, or cannot handle unseen accents and accent-unspecific standard speech. In this work, we aim to improve multi-accent speech recognition in the end-to-end (E2E) framework with a novel layer-wise adaptation architecture. Firstly, we propose a robust deep accent representation learning architecture to obtain accurate accent embedding, and some advanced schemes are designed to further boost the quality of accent embeddings, including phone posteriorgram (PPG) feature, TTS based data augmentation in the training stage, test-time augmentation and multi-embedding fusion in the testing stage. Then, the layer-wise adaptation with accent embeddings is developed for fast accent adaptation in ASR, and two types of adapter layers are designed, including the gated adapter layer and multi-basis adapter layer. Compared to the usual two-pass adaptation, these adapter layers are injected between the ASR encoder layers to encode the accent information in ASR flexibly, and perform fast adaption on the corresponding speech accent. The experiments on Accent AESRC corpus show that the proposed deep accent representation learning can capture accurate accent knowledge, and get high performance on accent classification. The new layer-wise adaptation architecture with the accurate accent embedding outperforms the other traditional methods, and obtains consistent $\sim$15% relative word error rate (WER) reduction on all kinds of testing scenarios, including seen accents, unseen accents and accent-unspecific standard speech.

End-to-End Speech Recognition from Federated Acoustic Models

Acoustic Model Fusion for End-to-end Speech Recognition

Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model

Federated Learning with Differential Privacy for End-to-End Speech Recognition

Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASR

The Gift of Feedback: Improving ASR Model Quality by Learning from User Corrections through Federated Learning

End-to-End Dereverberation, Beamforming, and Speech Recognition in a Cocktail Party.

Modular End-to-End Automatic Speech Recognition Framework for Acoustic-to-Word Model

Federated Learning of Large ASR Models in the Real World

Leveraging Redundancy in Multiple Audio Signals for Far-Field Speech Recognition

Federated Representation Learning for Automatic Speech Recognition

Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus

A GDPR-compliant Ecosystem for Speech Recognition with Transfer, Federated, and Evolutionary Learning

Recent Advances in End-to-End Automatic Speech Recognition

End-to-End Speech Recognition: A Survey

Extremely Low Footprint End-to-End ASR System for Smart Device

Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition

Towards Fast and Accurate Streaming End-To-End ASR

Reliability-Based Large-Vocabulary Audio-Visual Speech Recognition

Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study

End-to-end Contextual Speech Recognition Using Class Language Models and a Token Passing Decoder.