Abstract:End-to-end speech recognition, such as attention based approaches, is an emerging and attractive topic in recent years. It has achieved comparable performance with the traditional speech recognition framework. Because end-to-end approaches integrate acoustic and linguistic information into one model, the perturbation in the acoustic level such as acoustic noise, could be easily propagated to the linguistic level. Thus improving model robustness in real application environments for these end-to-end systems is crucial. In this paper, in order to make the attention based end-to-end model more robust against noises, we formulate regulation of the objective function with adversarial training examples. Particularly two adversarial regularization techniques, the fast gradient-sign method and the local distributional smoothness method, are explored to improve noise robustness. Experiments on two publicly available Chinese Mandarin corpora, AISHELL-1 and AISHELL-2, show that adversarial regularization is an effective approach to improve robustness against noises for our attention-based models. Specifically, we obtained 18.4% relative character error rate (CER) reduction on the AISHELL-1 noisy test set. Even on the clean test set, we showed 16.7% relative improvement. As the training set increases and covers more environmental varieties, our proposed methods remain effective despite that the improvement shrinks. Training on the large AISHELL-2 training corpus and testing on the various AISHELL-2 test sets, we achieved 7.0%-12.2% relative error rate reduction. To our knowledge, this is the first successful application of adversarial regularization to sequence-to-sequence speech recognition systems.

Unsupervised Adaptation with Adversarial Dropout Regularization for Robust Speech Recognition

Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition

An Unsupervised Deep Domain Adaptation Approach for Robust Speech Recognition

Unsupervised Domain Adaptation for Robust Speech Recognition via Variational Autoencoder-Based Data Augmentation

Unsupervised Domain Adaptation Via Domain Adversarial Training for Speaker Recognition.

Adversarial Regularization for Attention Based End-to-End Robust Speech Recognition.

Unsupervised Domain Adaptation for Dysarthric Speech Detection via Domain Adversarial Training and Mutual Information Minimization

Disentangled Adversarial Domain Adaptation for Phonation Mode Detection in Singing and Speech

Self-Supervised Learning Based Domain Adaptation for Robust Speaker Verification

Learning Invariant Representation and Risk Minimized for Unsupervised Accent Domain Adaptation

Channel-Aware Domain-Adaptive Generative Adversarial Network for Robust Speech Recognition

Cross-lingual Text-independent Speaker Verification using Unsupervised Adversarial Discriminative Domain Adaptation

Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation

Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training

Joint Discriminator and Transfer Based Fast Domain Adaptation for End-To-End Speech Recognition

Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters

Batch Normalization based Unsupervised Speaker Adaptation for Acoustic Models

Unsupervised Regularization-Based Adaptive Training for Speech Recognition

Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

Adversarial Domain Adaptation for Speaker Verification Using Partially Shared Network.

Unsupervised domain adaptation for deep neural network based voice activity detection