Abstract:Speech separation is the task of extracting target speech while suppressing background interference components. In applications like video telephones, visual information about the target speaker is available, which can be leveraged for multi-speaker speech separation. Most previous multi-speaker separation methods are mainly based on convolutional or recurrent neural networks. Recently, Transformer-based Seq2Seq models have achieved state-of-the-art performance in various tasks, such as neural machine translation (NMT), automatic speech recognition (ASR), etc. Transformer has showed an advantage in modeling audio-visual temporal context by multi-head attention blocks through explicitly assigning attention weights. Besides, Transformer doesn't have any recurrent sub-networks, thus supporting parallelization of sequence computation. In this paper, we propose a novel speaker-independent audio-visual speech separation method based on Transformer, which can be flexibly applied to unknown number and identity of speakers. The model receives both audiovisual streams, including noisy spectrogram and speaker lip embeddings, and predicts a complex time-frequency mask for the corresponding target speaker. The model is made up by three main components: audio encoder, visual encoder and Transformer-based mask generator. Two different structures of encoders are investigated and compared, including ResNet-based and Transformer-based. The performance of the proposed method is evaluated in terms of source separation and speech quality metrics. The experimental results on the benchmark GRID dataset show the effectiveness of the method on speaker-independent separation task in multi-talker environments. The model generalizes well to unseen identities of speakers and noise types. Though only trained on 2-speaker mixtures, the model achieves reasonable performance when tested on 2-speaker and 3-speaker mixtures. Besides, the model still shows an advantage compared with previous audio-visual speech separation works.

Disentangled-Transformer: An Explainable End-to-End Automatic Speech Recognition Model with Speech Content-Context Separation

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation

Explainability of Speech Recognition Transformers Via Gradient-Based Attention Visualization

Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction

Disentangling Voice and Content with Self-Supervision for Speaker Recognition

SpeechTripleNet: End-to-End Disentangled Speech Representation Learning for Content, Timbre and Prosody

Investigating Speaker Embedding Disentanglement on Natural Read Speech

Speaker-Independent Audio-Visual Speech Separation Based on Transformer in Multi-Talker Environments

Resource-Efficient Separation Transformer

Transformer-Transducers for Code-Switched Speech Recognition

An empirical analysis of information encoded in disentangled neural speaker representations

SETransformer: Speech Enhancement Transformer

An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer

Transformer with Bidirectional Decoder for Speech Recognition

Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation

Explaining How Transformers Use Context to Build Predictions

TransMask: A Compact and Fast Speech Separation Model Based on Transformer

Disentangled Speech Representation Learning for One-Shot Cross-Lingual Voice Conversion Using SS-Vae