Abstract:We propose a novel speaker-dependent (SD) multi-condition (MC) training approach to joint learning of deep neural networks (DNNs) of acoustic models and an explicit speech separation structure for recognition of multi-talker mixed speech in a single-channel setting. First, an MC acoustic modeling framework is established to train a SD-DNN model in multi-talker scenarios. Such a recognizer significantly reduces the decoding complexity and improves the recognition accuracy over those using speaker-independent DNN models with a complicated joint decoding structure assuming the speaker identities in mixed speech are known. In addition, a SD regression DNN for mapping the acoustic features of mixed speech to the speech features of a target speaker is jointly trained with the SD-DNN based acoustic models. Experimental results on Speech Separation Challenge (SSC) small-vocabulary recognition show that the proposed approach under multi-condition training achieves an average word error rate (WER) of 3.8%, yielding a relative WER reduction of 65.1% from a top performance, DNN-based pre-processing only approach we proposed earlier under clean-condition training (Tu et al. 2016). Furthermore, the proposed joint training DNN framework generates a relative WER reduction of 13.2% from state-of-the-art systems under multi-condition training. Finally, the effectiveness of the proposed approach is also verified on the Wall Street Journal (WSJ0) task with medium-vocabulary continuous speech recognition in a simulated multi-talker setting.

Towards Neural Speaker Modeling in Multi-Party Conversation: the Task, Dataset, and Models.

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

A Persona-Based Neural Conversation Model

Speaker-Aware Discourse Parsing on Multi-Party Dialogues.

Training Multi-Speaker Neural Text-to-Speech Systems using Speaker-Imbalanced Speech Corpora

Automatic Text-based Personality Recognition on Monologues and Multiparty Dialogues Using Attentive Networks and Contextual Embeddings

A speaker-aware multiparty dialogue discourse parser with heterogeneous graph neural network

Hybrid Architecture Based on Fuzzy Classifier and Multiplayer Feed-Forward Neural Network for Speaker Identification

Multi-layer Attention Mechanism Based Speech Separation Model.

Enhanced Speaker-Aware Multi-Party Multi-Turn Dialogue Comprehension

A Speaker-Dependent Approach to Single-Channel Joint Speech Separation and Acoustic Modeling Based on Deep Neural Networks for Robust Recognition of Multi-Talker Speech

A survey of multi-party dialogue research based on deep learning

Training Multi-Task Adversarial Network for Extracting Noise-Robust Speaker Embedding

Human-Machine Conversation Based on Hybrid Neural Network

MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding

Joint speaker encoder and neural back-end model for fully end-to-end automatic speaker verification with multiple enrollment utterances

Collaborative Learning for Language and Speaker Recognition.

Adapting Self-Supervised Models to Multi-Talker Speech Recognition Using Speaker Embeddings

Collaborative Joint Training With Multitask Recurrent Model for Speech and Speaker Recognition.

Semi-supervised multi-channel speaker diarization with cross-channel attention

Acoustic Modeling for Multi-Array Conversational Speech Recognition in the Chime-6 Challenge