Abstract:Although keyword spotting (KWS) technologies have been successfully applied to some applications, most KWS systems have a common problem of noise-robustness when applied to real-world environments. Audio-visual keyword spotting (AVKWS) using both acoustic and visual information is a solution to complementarily solve the problem. Most existing audio-visual speech recognition (AVSR) systems extract geometric features as visual features, which heavily rely on accurate and reliable detection and tracking of facial feature points. To avoid this defect of geometric features, an appearance-based discriminative local spatial-temporal descriptor (disCLBP-TOP) is proposed in this paper, which devotes to extracting robust and discriminative patterns of interest. Besides, a parallel two-step recognition based on both acoustic and visual keyword searching and re-scoring is conducted, which complementarily makes the best of two modalities under different noisy conditions. Adaptive weights for decision fusion are generated using a sigmoid function based on reliabilities of the two modalities, capable of adapting to various noisy conditions. Experiments show that our proposed parallel AVKWS strategy based on decision fusion significantly improves the noise robustness and attains better performance than feature fusion based audio-visual spotter. Additionally, disCLBP-TOP shows more competitive performance than CLBP-TOP.

A Novel Discriminative Score Calibration Method for Keyword Search

Calibration of Word Posterior Estimation in Confusion Networks for Keyword Search

Experimental Investigation into Alignment-based Acoustic Confidence Measures in Keyword Verification for Mandarin Speech

Improved Keyword Spotting System by Optimizing Posterior Confidence Measure Vector Using Feed-Forward Neural Network.

Keyword Confidence Evaluation Algorithm Based on Word Activation Forces

Improved System Fusion for Keyword Search

Auc Optimization Based Confidence Measure for Keyword Spotting

A Rescoring Approach for Keyword Search Using Lattice Context Information.

Improving keyword search by query expansion in a probabilistic framework

Integrate Document Ranking Information into Confidence Measure Calculation for Spoken Term Detection.

Keyword Spotting Based on Hypothesis Boundary Realignment and State-Level Confidence Weighting

Word Spotting Based on a Posterior Measure of Keyword Confidence

A Novel Re-weighted CTC Loss for Data Imbalance in Speech Keyword Spotting

Mandarin keyword spotting using syllable based confidence features and SVM

An lstm-ctc based verification system for proxy-word based oov keyword search

Phoneme-Level Contrastive Learning for User-Defined Keyword Spotting with Flexible Enrollment

Multi-class AUC Optimization for Robust Small-footprint Keyword Spotting with Limited Training Data

Keyword-specific normalization based keyword spotting for spontaneous speech

Keyword Spotting Based on Syllable Confusion Network.

A Two-Step Keyword Spotting Method Based on Context-Dependent a Posteriori Probability

Audio-visual Keyword Spotting for Mandarin Based on Discriminative Local Spatial-Temporal Descriptors.