Abstract:The application of data-intensive automatic speech recognition (ASR) technologies to dysarthric and elderly adult speech is confronted by their mismatch against healthy and nonaged voices, data scarcity and large speaker-level variability. To this end, this paper proposes two novel data-efficient methods to learn homogeneous dysarthric and elderly speaker-level features for rapid, on-the-fly test-time adaptation of DNN/TDNN and Conformer ASR models. These include: 1) speaker-level variance-regularized spectral basis embedding (VR-SBE) features that exploit a special regularization term to enforce homogeneity of speaker features in adaptation; and 2) feature-based learning hidden unit contributions (f-LHUC) transforms that are conditioned on VR-SBE features. Experiments are conducted on four tasks across two languages: the English UASpeech and TORGO dysarthric speech datasets, the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech corpora. The proposed on-the-fly speaker adaptation techniques consistently outperform baseline iVector and xVector adaptation by statistically significant word or character error rate reductions up to 5.32% absolute (18.57% relative) and batch-mode LHUC speaker adaptation by 2.24% absolute (9.20% relative), while operating with real-time factors speeding up to 33.6 times against xVectors during adaptation. The efficacy of the proposed adaptation techniques is demonstrated in a comparison against current ASR technologies including SSL pre-trained systems on UASpeech, where our best system produces a state-of-the-art WER of 23.33%. Analyses show VR-SBE features and f-LHUC transforms are insensitive to speaker-level data quantity in testtime adaptation. T-SNE visualization reveals they have stronger speaker-level homogeneity than baseline iVectors, xVectors and batch-mode LHUC transforms.

Linguistic Feedback Supports Rapid Adaptation to Acoustically Degraded Speech

Automatic Auditory Streaming Restores Missing Temporal Modulations in Echoic Speech

Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech

Survey of Speaker Adaptation Methods in Speech Recognition

Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation

On-the-Fly Feature Based Rapid Speaker Adaptation for Dysarthric and Elderly Speech Recognition

Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

Listen, Attend, Spell and Adapt: Speaker Adapted Sequence-to-Sequence ASR

Speaker normalization and adaptation techniques in automatic pronunciation evaluation

Investigating Performance of the Discriminative Methods for Long-Term Speaker Adaptation

Rapid Adaptation For Deep Neural Networks Through Multi-Task Learning

Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems

An Active Learning Approach to Task Adaptation.

Adaptation to novel foreign-accented speech and retention of benefit following training: Influence of aging and hearing loss

Modeling speech adaptation to altered sensory feedback through continuous learning of internal sensory predictions

Integrated Adaptation with Multi-Factor Joint-Learning for Far-Field Speech Recognition

Improvement of Acoustic Models Fused with Lip Visual Information for Low-Resource Speech

Learning Fast Adaptation on Cross-Accented Speech Recognition

Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation

No More Mumbles: Enhancing Robot Intelligibility through Speech Adaptation

Speech perception under adverse conditions: insights from behavioral, computational, and neuroscience research