Abstract:Speech separation is the task of extracting target speech while suppressing background interference components. In applications like video telephones, visual information about the target speaker is available, which can be leveraged for multi-speaker speech separation. Most previous multi-speaker separation methods are mainly based on convolutional or recurrent neural networks. Recently, Transformer-based Seq2Seq models have achieved state-of-the-art performance in various tasks, such as neural machine translation (NMT), automatic speech recognition (ASR), etc. Transformer has showed an advantage in modeling audio-visual temporal context by multi-head attention blocks through explicitly assigning attention weights. Besides, Transformer doesn't have any recurrent sub-networks, thus supporting parallelization of sequence computation. In this paper, we propose a novel speaker-independent audio-visual speech separation method based on Transformer, which can be flexibly applied to unknown number and identity of speakers. The model receives both audiovisual streams, including noisy spectrogram and speaker lip embeddings, and predicts a complex time-frequency mask for the corresponding target speaker. The model is made up by three main components: audio encoder, visual encoder and Transformer-based mask generator. Two different structures of encoders are investigated and compared, including ResNet-based and Transformer-based. The performance of the proposed method is evaluated in terms of source separation and speech quality metrics. The experimental results on the benchmark GRID dataset show the effectiveness of the method on speaker-independent separation task in multi-talker environments. The model generalizes well to unseen identities of speakers and noise types. Though only trained on 2-speaker mixtures, the model achieves reasonable performance when tested on 2-speaker and 3-speaker mixtures. Besides, the model still shows an advantage compared with previous audio-visual speech separation works.

Concurrent Speaker Detection: A multi-microphone Transformer-Based Approach

Audio-Visual Approach For Multimodal Concurrent Speaker Detection

CST: Automatic Modulation Recognition Method by Convolution Transformer on Temporal Continuity Features

Focal Kl-Divergence Based Dilated Convolutional Neural Networks for Co-Channel Speaker Identification.

Monaural Multi-Speaker Speech Separation Using Efficient Transformer Model

MEConformer: Highly representative embedding extractor for speaker verification via incorporating selective convolution into deep speaker encoder

Multi-microphone fusion for detection of speech and acoustic events in smart spaces

A Transformer Model for Segmentation, Classification, and Caller Identification of Marmoset Vocalization

Backchannel Detection and Agreement Estimation from Video with Transformer Networks

Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanism

Improving Transformer-based Networks With Locality For Automatic Speaker Verification

PCNN: A Lightweight Parallel Conformer Neural Network for Efficient Monaural Speech Enhancement

Speaker Change Detection for Transformer Transducer ASR

Look&listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement

Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition

Speaker-Independent Audio-Visual Speech Separation Based on Transformer in Multi-Talker Environments

Multi-Microphone Speech Emotion Recognition using the Hierarchical Token-semantic Audio Transformer Architecture

A Spatial Long-Term Iterative Mask Estimation Approach for Multi-Channel Speaker Diarization and Speech Recognition.

Binaural Speech Enhancement Using Deep Complex Convolutional Transformer Networks

Explore Relative and Context Information with Transformer for Joint Acoustic Echo Cancellation and Speech Enhancement

End-to-End Multi-speaker Speech Recognition with Transformer.