Abstract:The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves a 2.79% false rejection rate and a 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge.

Keyword-Specific Acoustic Model Pruning for Open-Vocabulary Keyword Spotting

U2-KWS: Unified Two-pass Open-vocabulary Keyword Spotting with Keyword Bias

Weight-importance sparse training in keyword spotting

Keyword-specific normalization based keyword spotting for spontaneous speech

An Improved Point Process Models for Spotting Keywords in Continuous Speech

A Multitask Training Approach to Enhance Whisper with Open-Vocabulary Keyword Spotting

DCCRN-KWS: an audio bias based model for noise robust small-footprint keyword spotting

Focal Loss And Double-Edge-Triggered Detector For Robust Small-Footprint Keyword Spotting

A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting

Keyword Spotting Based on Restricting Model and Acoustic Confidence Measure

A New Keyword Spotting Approach for Spontaneous Mandarin Speech

CaTT-KWS: A Multi-stage Customized Keyword Spotting Framework Based on Cascaded Transducer-Transformer.

A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning

VE-KWS: Visual Modality Enhanced End-to-End Keyword Spotting

Web-based keyword adapted Language Modeling for Keyword Spotting

Synth4Kws: Synthesized Speech for User Defined Keyword Spotting in Low Resource Environments

End-to-End Speech Keyword Spotting System

Exploring Representation Learning for Small-Footprint Keyword Spotting

NS-KWS: joint optimization of near-sensor processing architecture and low-precision GRU for always-on keyword spotting

Open-vocabulary Keyword-spotting with Adaptive Instance Normalization