Abstract:The application of data-intensive automatic speech recognition (ASR) technologies to dysarthric and elderly adult speech is confronted by their mismatch against healthy and nonaged voices, data scarcity and large speaker-level variability. To this end, this paper proposes two novel data-efficient methods to learn homogeneous dysarthric and elderly speaker-level features for rapid, on-the-fly test-time adaptation of DNN/TDNN and Conformer ASR models. These include: 1) speaker-level variance-regularized spectral basis embedding (VR-SBE) features that exploit a special regularization term to enforce homogeneity of speaker features in adaptation; and 2) feature-based learning hidden unit contributions (f-LHUC) transforms that are conditioned on VR-SBE features. Experiments are conducted on four tasks across two languages: the English UASpeech and TORGO dysarthric speech datasets, the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora. The proposed on-the-fly speaker adaptation techniques consistently outperform baseline iVector and xVector adaptation by statistically significant word or character error rate reductions up to 5.32% absolute (18.57% relative) and batch-mode LHUC speaker adaptation by 2.24% absolute (9.20% relative), while operating with real-time factors speeding up to 33.6 times against xVectors during adaptation. The efficacy of the proposed adaptation techniques is demonstrated in a comparison against current ASR technologies including SSL pre-trained systems on UASpeech, where our best system produces a state-of-the-art WER of 23.33%. Analyses show VR-SBE features and f-LHUC transforms are insensitive to speaker-level data quantity in testtime adaptation. T-SNE visualization reveals they have stronger speaker-level homogeneity than baseline iVectors, xVectors and batch-mode LHUC transforms.

Listen, Attend, Spell and Adapt: Speaker Adapted Sequence-to-Sequence ASR

Linguistic Feedback Supports Rapid Adaptation to Acoustically Degraded Speech

Speaker Normalization Training and Adaptation for Speech Recognition

Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR

Survey of Speaker Adaptation Methods in Speech Recognition

Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

Rapid Speaker Adaptation Based on D-code Extracted from BLSTM-RNN in LVCSR

A Unified Speaker Adaptation Method for Speech Synthesis using Transcribed and Untranscribed Speech with Backpropagation

Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation

Investigating Performance of the Discriminative Methods for Long-Term Speaker Adaptation

KL-Divergence Regularized Deep Neural Network Adaptation for Low-Resource Speaker-Dependent Speech Enhancement

Linear Networks Based Speaker Adaptation for Speech Synthesis

Unsupervised Speaker Adaptation of Blstm-Rnn for Lvcsr Based on Speaker Code

Speaker Adaptation for End-to-End CTC Models.

Speaker Adaptation of Hybrid NN/HMM Model for Speech Recognition Based on Singular Value Decomposition

Speaker adaptation using maximum likelihood model interpolation

Codebook-Based Speaker Adaptation

Speaker adaptive bottleneck features extraction for LVCSR based on discriminative learning of speaker codes

Online Speaker Adaptation for LVCSR Based on Attention Mechanism

Unsupervised Speaker Adaptation Of Deep Neural Network Based On The Combination Of Speaker Codes And Singular Value Decomposition For Speech Recognition

On-the-Fly Feature Based Rapid Speaker Adaptation for Dysarthric and Elderly Speech Recognition