Abstract:The application of data-intensive automatic speech recognition (ASR) technologies to dysarthric and elderly adult speech is confronted by their mismatch against healthy and nonaged voices, data scarcity and large speaker-level variability. To this end, this paper proposes two novel data-efficient methods to learn homogeneous dysarthric and elderly speaker-level features for rapid, on-the-fly test-time adaptation of DNN/TDNN and Conformer ASR models. These include: 1) speaker-level variance-regularized spectral basis embedding (VR-SBE) features that exploit a special regularization term to enforce homogeneity of speaker features in adaptation; and 2) feature-based learning hidden unit contributions (f-LHUC) transforms that are conditioned on VR-SBE features. Experiments are conducted on four tasks across two languages: the English UASpeech and TORGO dysarthric speech datasets, the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora. The proposed on-the-fly speaker adaptation techniques consistently outperform baseline iVector and xVector adaptation by statistically significant word or character error rate reductions up to 5.32% absolute (18.57% relative) and batch-mode LHUC speaker adaptation by 2.24% absolute (9.20% relative), while operating with real-time factors speeding up to 33.6 times against xVectors during adaptation. The efficacy of the proposed adaptation techniques is demonstrated in a comparison against current ASR technologies including SSL pre-trained systems on UASpeech, where our best system produces a state-of-the-art WER of 23.33%. Analyses show VR-SBE features and f-LHUC transforms are insensitive to speaker-level data quantity in testtime adaptation. T-SNE visualization reveals they have stronger speaker-level homogeneity than baseline iVectors, xVectors and batch-mode LHUC transforms.

Reference Point Alignment Frequency Warp Method for Speaker Adaptation

Reference Point Alignment Frequency Warpmethod For Speaker Adaptation

Pitch mean based frequency warping

Speaker adaptation based on combination of MAP estimation and weighted neighbor regression

Speaker Adaptation with MAP Estimation and Weighted Neighbor Regression

Frequency Domain Correspondence For Speaker Normalization

Reference-Based Speech Enhancement Via Feature Alignment and Fusion Network

A New Subspace Based Speaker Adaptation Method

Robust Speech Recognition Based on Spectral Adjusting and Warping

Online Speaker Adaptation for WaveNet-based Neural Vocoders

Speech Recognition Using Speaker Adaptation by System Parameter Transformation.

Speaker Adaptation for Telephony Data Using Speaker Clustering

Spoken Content and Voice Factorization for Few-Shot Speaker Adaptation

Phoneme Dependent Speaker Embedding And Model Factorization For Multi-Speaker Speech Synthesis And Adaptation

Speaker-independent Training and Adaptation of Neural Vocoders

Speaker adaptation method using principal curves algorithm

Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

A Combined Speaker Adaptation Method for Mandarin Speech Recognition

Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation

Dynamic Speaker Representations Adjustment and Decoder Factorization for Speaker Adaptation in End-to-End Speech Synthesis

Speaker Normalization Training and Adaptation for Speech Recognition