Abstract:Keyword spotting remains a challenge when applied to real-world environments with dramatically changing noise. In recent studies, audio-visual integration methods have demonstrated superiorities since visual speech is not influenced by acoustic noise. However, for visual speech recognition, individual utterance mannerisms can lead to confusion and false recognition. To solve this problem, a novel lip descriptor is presented involving both geometry-based and appearance-based features in this paper. Specifically, a set of geometry-based features is proposed based on an advanced facial landmark localization method. In order to obtain robust and discriminative representation, a spatiotemporal lip feature is put forward concerning similarities among textons and mapping the feature to intra-class subspace. Moreover, a parallel two-step keyword spotting strategy based on decision fusion is proposed in order to make the best use of audio-visual speech and adapt to diverse noise conditions. Weights generated using a neural network combine acoustic and visual contributions. Experimental results on the OuluVS dataset and PKU-AV dataset demonstrate that the proposed lip descriptor shows competitive performance compared to the state of the art. Additionally, the proposed audio-visual keyword spotting (AV-KWS) method based on decision-level fusion significantly improves the noise robustness and attains better performance than feature-level fusion, which is also capable of adapting to various noisy conditions.

A NOVEL TWO-LEVEL ARCHITECTURE PLUS CONFIDENCE MEASURES FOR A KEYWORD SPOTTING SYSTEM

LEXICAL ACCESS-BASED CONFIDENCE MEASURE FOR A SPANISH KEYWORD SPOTTING SYSTEM

A New Framework for Large Vocabulary Keyword Spotting Using Two-Pass Confidence Measure

Experimental Investigation into Alignment-based Acoustic Confidence Measures in Keyword Verification for Mandarin Speech

An Evolutionary Confidence Measure for Spotting Words in Speech Recognition

Word Spotting Based on a Posterior Measure of Keyword Confidence

Improved Keyword Spotting System by Optimizing Posterior Confidence Measure Vector Using Feed-Forward Neural Network.

Keyword Spotting Based on Hypothesis Boundary Realignment and State-Level Confidence Weighting

Keyword-specific normalization based keyword spotting for spontaneous speech

A Two-Step Keyword Spotting Method Based on Context-Dependent a Posteriori Probability

A New Keyword Spotting Approach for Spontaneous Mandarin Speech

A Keyword Spotting Method

A Two-Step Keyword Spotting Method Using Fuzzy Search Algorithm

U2-KWS: Unified Two-pass Open-vocabulary Keyword Spotting with Keyword Bias

A Novel Discriminative Score Calibration Method for Keyword Search

A Dialogue System Based on Keyword Spotting Technology

Keyword Spotting Based on Phoneme Confusion Matrix

Audio-visual Keyword Spotting Based on Adaptive Decision Fusion under Noisy Conditions for Human-Robot Interaction.

A Novel Lip Descriptor for Audio-Visual Keyword Spotting Based on Adaptive Decision Fusion

Keyword Spotting Simplified: A Segmentation-Free Approach using Character Counting and CTC re-scoring

A Posterior Probability-Based System Hybridisation and Combination for Spoken Term Detection