Abstract:Accurate recognition of cocktail party speech containing overlapping speakers, noise and reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all system components is proposed in this paper. The efficacy of the video input is consistently demonstrated in mask-based MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end and Conformer ASR back-end. Audio-visual integrated front-end architectures performing speech separation and dereverberation in a pipelined or joint fashion via mask-based WPD are investigated. The error cost mismatch between the speech enhancement front-end and ASR back-end components is minimized by end-to-end jointly fine-tuning using either the ASR cost function alone, or its interpolation with the speech enhancement loss. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset. The proposed audio-visual multi-channel speech separation, dereverberation and recognition systems consistently outperformed the comparable audio-only baseline by 9.1% and 6.2% absolute (41.7% and 36.0% relative) word error rate (WER) reductions. Consistent speech enhancement improvements were also obtained on PESQ, STOI and SRMR scores.

Combining Log-Spectral Domain Compensation with MVA Feature Post-Processing for Robust Speech Recognition

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Combining Eigenvoice Speaker Modeling And Vts-Based Environment Compensation For Robust Speech Recognition

Combining Noise Compensation and Missing-Feature Decoding for Large Vocabulary Speech Recognition in Noise

Replacing Uncertainty Decoding with Subband Re-Estimation for Large Vocabulary Speech Recognition in Noise.

An Improved VTS Feature Compensation Using Mixture Models of Distortion and IVN Training for Noisy Speech Recognition

A Feature Compensation Approach Using Piecewise Linear Approximation of an Explicit Distortion Model for Noisy Speech Recognition

An Efficient Robust Asr System Based On The Combination Of Speech Enhancement And Hmm Adaptation

Leveraging Joint Spectral and Spatial Learning with MAMBA for Multichannel Speech Enhancement

An Algorithm of Model Compensation Based on the Estimation of Additive Noise and Channel Function for Speech Recognition

Residual Noise Compensation For Robust Speech Recognition In Nonstationary Noise

Ivn-Based Joint Training of Gmm and Hmms Using an Improved Vts-Based Feature Compensation for Noisy Speech Recognition

Robust Speech Recognition Based on Spectral Adjusting and Warping

Multi-Objective Learning and Mask-Based Post-Processing for Deep Neural Network Based Speech Enhancement

Robust Log-Energy Estimation and Its Dynamic Change Enhancement for In-car Speech Recognition

Compensation of Speech Enhancement Distortion for Robust Speech Recognition

Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition

MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition

Robust Speaker Recognition in Cross-Channel Condition Based on Gaussian Mixture Model

Audio-visual multi-channel speech separation, dereverberation and recognition

A VTS-based Feature Compensation Approach to Noisy Speech Recognition Using Mixture Models of Distortion