Abstract:The application of data-intensive automatic speech recognition (ASR) technologies to dysarthric and elderly adult speech is confronted by their mismatch against healthy and nonaged voices, data scarcity and large speaker-level variability. To this end, this paper proposes two novel data-efficient methods to learn homogeneous dysarthric and elderly speaker-level features for rapid, on-the-fly test-time adaptation of DNN/TDNN and Conformer ASR models. These include: 1) speaker-level variance-regularized spectral basis embedding (VR-SBE) features that exploit a special regularization term to enforce homogeneity of speaker features in adaptation; and 2) feature-based learning hidden unit contributions (f-LHUC) transforms that are conditioned on VR-SBE features. Experiments are conducted on four tasks across two languages: the English UASpeech and TORGO dysarthric speech datasets, the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora. The proposed on-the-fly speaker adaptation techniques consistently outperform baseline iVector and xVector adaptation by statistically significant word or character error rate reductions up to 5.32% absolute (18.57% relative) and batch-mode LHUC speaker adaptation by 2.24% absolute (9.20% relative), while operating with real-time factors speeding up to 33.6 times against xVectors during adaptation. The efficacy of the proposed adaptation techniques is demonstrated in a comparison against current ASR technologies including SSL pre-trained systems on UASpeech, where our best system produces a state-of-the-art WER of 23.33%. Analyses show VR-SBE features and f-LHUC transforms are insensitive to speaker-level data quantity in testtime adaptation. T-SNE visualization reveals they have stronger speaker-level homogeneity than baseline iVectors, xVectors and batch-mode LHUC transforms.

Rapid Speaker Adaptation Using Multi-Stream Structural Maximum Likelihood Eigenspace Mapping

Speaker adaptation using maximum likelihood model interpolation

Agmma: A Novel Incremental Adaptation Method And Its Application To Speaker Recognition

Rapid Feature Space MLLR Speaker Adaptation with Bilinear Models

Rapid discriminative acoustic model based on eigenspace mapping for fast speaker adaptation

Rapid Feature Space Mllr Speaker Adaptation for Deep Neural Network Acoustic Modeling

A Speaker Adaptation Algorithm Based on Matrix Linear Interpolation

SVM Based Speaker Selection Using GMM Supervector for Rapid Speaker Adaptation

MAP-based Speaker Adaptation in Speech Synthesis

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation

Rapid Speaker Adaptation Based on D-code Extracted from BLSTM-RNN in LVCSR

A New Subspace Based Speaker Adaptation Method

Discriminative Speaker Adaptation with Eigenvoices

Improving Online Incremental Speaker Adaptation with Eigen Feature Space MLLR.

Dynamic Speaker Selected Training for Rapid Speaker Adaptation

Speaker Adaptation and Adaptive Training for Jointly Optimised Tandem Systems.

Direct adaptation of hybrid DNN/HMM model for fast speaker adaptation in LVCSR based on speaker code

Model Dimensionality Selection in Bilinear Transformation for Feature Space MLLR Rapid Speaker Adaptation

On-the-Fly Feature Based Rapid Speaker Adaptation for Dysarthric and Elderly Speech Recognition

Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation

UBM Based Speaker Selection and Model Re-Estimation for Speaker Adaptation