Abstract:Although end-to-end (E2E) trainable automatic speech recognition (ASR) has shown great success by jointly learning acoustic and linguistic information, it still suffers from the effect of domain shifts, thus limiting potential applications. The E2E ASR model implicitly learns an internal language model (LM) which characterises the training distribution of the source domain, and the E2E trainable nature makes the internal LM difficult to adapt to the target domain with text-only data To solve this problem, this paper proposes decoupled structures for attention-based encoder-decoder (Decoupled-AED) and neural transducer (Decoupled-Transducer) models, which can achieve flexible domain adaptation in both offline and online scenarios while maintaining robust intra-domain performance. To this end, the acoustic and linguistic parts of the E2E model decoder (or prediction network) are decoupled, making the linguistic component (i.e. internal LM) replaceable. When encountering a domain shift, the internal LM can be directly replaced during inference by a target-domain LM, without re-training or using domain-specific paired speech-text data. Experiments for E2E ASR models trained on the LibriSpeech-100h corpus showed that the proposed decoupled structure gave 15.1% and 17.2% relative word error rate reductions on the TED-LIUM 2 and AESRC2020 corpora while still maintaining performance on intra-domain data.

Factorized AED: Factorized Attention-Based Encoder-Decoder for Text-Only Domain Adaptive ASR

Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation

Text Only Domain Adaptation with Phoneme Guided Data Splicing for End-to-End Speech Recognition

Efficient Decoding Self-Attention for End-to-end Speech Synthesis

Decoupled Structure for Improved Adaptability of End-to-End Models

Efficient Text-Only Domain Adaptation For CTC-Based ASR.

Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks

Optimizing Alignment of Speech and Language Latent Spaces for End-To-End Speech Recognition and Understanding.

Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems

Text-conditioned Transformer for Automatic Pronunciation Error Detection

Integrated Adaptation with Multi-Factor Joint-Learning for Far-Field Speech Recognition

Improved Factorized Neural Transducer Model For text-only Domain Adaptation

Joint Discriminator and Transfer Based Fast Domain Adaptation for End-To-End Speech Recognition

LongFNT: Long-form Speech Recognition with Factorized Neural Transducer

Efficient Long-Form Speech Recognition for General Speech In-Context Learning

Rethinking Speech Recognition with A Multimodal Perspective via Acoustic and Semantic Cooperative Decoding

Text-only Domain Adaptation using Unified Speech-Text Representation in Transducer

Dynamic Speaker Representations Adjustment and Decoder Factorization for Speaker Adaptation in End-to-End Speech Synthesis

USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder

Label-Synchronous Neural Transducer for Adaptable Online E2E Speech Recognition

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation