Abstract:Speech separation is the task of extracting target speech while suppressing background interference components. In applications like video telephones, visual information about the target speaker is available, which can be leveraged for multi-speaker speech separation. Most previous multi-speaker separation methods are mainly based on convolutional or recurrent neural networks. Recently, Transformer-based Seq2Seq models have achieved state-of-the-art performance in various tasks, such as neural machine translation (NMT), automatic speech recognition (ASR), etc. Transformer has showed an advantage in modeling audio-visual temporal context by multi-head attention blocks through explicitly assigning attention weights. Besides, Transformer doesn't have any recurrent sub-networks, thus supporting parallelization of sequence computation. In this paper, we propose a novel speaker-independent audio-visual speech separation method based on Transformer, which can be flexibly applied to unknown number and identity of speakers. The model receives both audiovisual streams, including noisy spectrogram and speaker lip embeddings, and predicts a complex time-frequency mask for the corresponding target speaker. The model is made up by three main components: audio encoder, visual encoder and Transformer-based mask generator. Two different structures of encoders are investigated and compared, including ResNet-based and Transformer-based. The performance of the proposed method is evaluated in terms of source separation and speech quality metrics. The experimental results on the benchmark GRID dataset show the effectiveness of the method on speaker-independent separation task in multi-talker environments. The model generalizes well to unseen identities of speakers and noise types. Though only trained on 2-speaker mixtures, the model achieves reasonable performance when tested on 2-speaker and 3-speaker mixtures. Besides, the model still shows an advantage compared with previous audio-visual speech separation works.

Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation

Listen, Watch and Understand at the Cocktail Party: Audio-Visual-Contextual Speech Separation

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model

Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition

Speaker-Independent Audio-Visual Speech Separation Based on Transformer in Multi-Talker Environments

Audio-visual speech separation based on joint feature representation with cross-modal attention

Monaural Multi-Speaker Speech Separation Using Efficient Transformer Model

Audio-Visual Multi-Talker Speech Recognition in a Cocktail Party.

Multi-Stream Gated and Pyramidal Temporal Convolutional Neural Networks for Audio-Visual Speech Separation in Multi-Talker Environments

Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation

Listening and Grouping: an Online Autoregressive Approach for Monaural Speech Separation

Weakly-supervised Audio Separation via Bi-modal Semantic Similarity

Listen and Look: Audio–Visual Matching Assisted Speech Source Separation

Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning

Audio-visual multi-channel speech separation, dereverberation and recognition

Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanism

Speech Separation based on Contrastive Learning and Deep Modularization

Deep Audio-Visual Speech Separation with Attention Mechanism

Look&listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement