Abstract:Due to the dynamic nature of human language, automatic speech recognition (ASR) systems need to continuously acquire new vocabulary. Out-Of-Vocabulary (OOV) words, such as trending words and new named entities, pose problems to modern ASR systems that require long training times to adapt their large numbers of parameters. Different from most previous research focusing on language model post-processing, we tackle this problem on an earlier processing level and eliminate the bias in acoustic modeling to recognize OOV words acoustically. We propose to generate OOV words using text-to-speech systems and to rescale losses to encourage neural networks to pay more attention to OOV words. Specifically, we enlarge the classification loss used for training neural networks' parameters of utterances containing OOV words (sentence-level), or rescale the gradient used for back-propagation for OOV words (word-level), when fine-tuning a previously trained model on synthetic audio. To overcome catastrophic forgetting, we also explore the combination of loss rescaling and model regularization, i.e. L2 regularization and elastic weight consolidation (EWC). Compared with previous methods that just fine-tune synthetic audio with EWC, the experimental results on the LibriSpeech benchmark reveal that our proposed loss rescaling approach can achieve significant improvement on the recall rate with only a slight decrease on word error rate. Moreover, word-level rescaling is more stable than utterance-level rescaling and leads to higher recall rates and precision on OOV word recognition. Furthermore, our proposed combined loss rescaling and weight consolidation methods can support continual learning of an ASR system.

Listen, Know and Spell: Knowledge-Infused Subword Modeling for Improving ASR Performance of OOV Named Entities

Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition

A Discriminative Entity-Aware Language Model for Virtual Assistants

Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition

LM-assisted keyword biasing with Aho-Corasick algorithm for Transducer-based ASR

Server-side Rescoring of Spoken Entity-centric Knowledge Queries for Virtual Assistants

Sequential Editing for Lifelong Training of Speech Recognition Models

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting

OOVs in the Spotlight: How to Inflect them?

Neural Named Entity Recognition from Subword Units

Speech Recognition Rescoring with Large Speech-Text Foundation Models

Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

Enhancing Out-of-Vocabulary Performance of Indian TTS Systems for Practical Applications through Low-Effort Data Strategies

Enhancing Large Language Models with Knowledge Graphs for Robust Question Answering

Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking

Retrieval Augmented Correction of Named Entity Speech Recognition Errors

Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection

Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answering

Spell my name: keyword boosted speech recognition

Initial Decoding with Minimally Augmented Language Model for Improved Lattice Rescoring in Low Resource ASR