Abstract:Fast adaptation of deep neural networks (DNN) is an important research topic in deep learning. In this paper, we have proposed a general adaptation scheme for DNN based on discriminant condition codes, which are directly fed to various layers of a pre-trained DNN through a new set of connection weights. Moreover, we present several training methods to learn connection weights from training data as well as the corresponding adaptation methods to learn new condition code from adaptation data for each new test condition. In this work, the fast adaptation scheme is applied to supervised speaker adaptation in speech recognition based on either frame-level cross-entropy or sequence-level maximum mutual information training criterion. We have proposed three different ways to apply this adaptation scheme based on the so-called speaker codes: i) Nonlinear feature normalization in feature space; ii) Direct model adaptation of DNN based on speaker codes; iii) Joint speaker adaptive training with speaker codes. We have evaluated the proposed adaptation methods in two standard speech recognition tasks, namely TIMIT phone recognition and large vocabulary speech recognition in the Switchboard task. Experimental results have shown that all three methods are quite effective to adapt large DNN models using only a small amount of adaptation data. For example, the Switchboard results have shown that the proposed speaker-code-based adaptation methods may achieve up to 8-10% relative error reduction using only a few dozens of adaptation utterances per speaker. Finally, we have achieved very good performance in Switchboard (12.1% in WER) after speaker adaptation using sequence training criterion, which is very close to the best performance reported in this task (“Deep convolutional neural networks for LVCSR,” T. N. Sainath , Proc. IEEE Acoust., Speech, Signal Process., 2013).

Low-Rank Plus Diagonal Adaptation For Deep Neural Networks

Extended Low-Rank Plus Diagonal Adaptation for Deep and Recurrent Neural Networks.

Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach

Investigating Online Low-Footprint Speaker Adaptation Using Generalized Linear Regression and Click-Through Data.

Fast Adaptation of Deep Neural Network Based on Discriminant Codes for Speech Recognition

Speaker Adaptation of Hybrid NN/HMM Model for Speech Recognition Based on Singular Value Decomposition

Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices

Rapid Adaptation For Deep Neural Networks Through Multi-Task Learning

The Expressive Power of Low-Rank Adaptation

Linear Networks Based Speaker Adaptation for Speech Synthesis

Low-Rank Interconnected Adaptation across Layers

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

Unsupervised Speaker Adaptation Of Deep Neural Network Based On The Combination Of Speaker Codes And Singular Value Decomposition For Speech Recognition

Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation

Derivative-Free Optimization for Low-Rank Adaptation in Large Language Models

Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning

LoTR: Low Tensor Rank Weight Adaptation

Stochastic Gradient Methods with Block Diagonal Matrix Adaptation

AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training