Abstract:Due to the dynamic nature of human language, automatic speech recognition (ASR) systems need to continuously acquire new vocabulary. Out-Of-Vocabulary (OOV) words, such as trending words and new named entities, pose problems to modern ASR systems that require long training times to adapt their large numbers of parameters. Different from most previous research focusing on language model post-processing, we tackle this problem on an earlier processing level and eliminate the bias in acoustic modeling to recognize OOV words acoustically. We propose to generate OOV words using text-to-speech systems and to rescale losses to encourage neural networks to pay more attention to OOV words. Specifically, we enlarge the classification loss used for training neural networks' parameters of utterances containing OOV words (sentence-level), or rescale the gradient used for back-propagation for OOV words (word-level), when fine-tuning a previously trained model on synthetic audio. To overcome catastrophic forgetting, we also explore the combination of loss rescaling and model regularization, i.e. L2 regularization and elastic weight consolidation (EWC). Compared with previous methods that just fine-tune synthetic audio with EWC, the experimental results on the LibriSpeech benchmark reveal that our proposed loss rescaling approach can achieve significant improvement on the recall rate with only a slight decrease on word error rate. Moreover, word-level rescaling is more stable than utterance-level rescaling and leads to higher recall rates and precision on OOV word recognition. Furthermore, our proposed combined loss rescaling and weight consolidation methods can support continual learning of an ASR system.

Fast Oov Words Incorporation Using Structured Word Embeddings for Neural Network Language Model.

Improve Word Embedding Using Both Writing and Pronunciation.

Recurrent Neural Network Language Model With Structured Word Embeddings For Speech Recognition

Deep learning models for representing out-of-vocabulary words

OOV Words Extension for Modular Neural Acoustics-to-Word Model

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition.

Now It Sounds Like You: Learning Personalized Vocabulary On Device

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition

Structured Word Embedding For Low Memory Neural Network Language Model

Estimator Vectors: OOV Word Embeddings based on Subword and Context Clue Estimates

Federated Learning Of Out-Of-Vocabulary Words

Enriching Rare Word Representations in Neural Language Models by Embedding Matrix Augmentation

OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining

From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding

Improving Pre-Trained Multilingual Model with Vocabulary Expansion

Learning Task-Specific Representation for Novel Words in Sequence Labeling.

Vision-Language Adaptive Mutual Decoder for OOV-STR

Compressing Neural Language Models by Sparse Word Representations

Explicit Word Density Estimation for Language Modelling

Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus