Abstract:Speaker diarization consists of assigning speech signals to people engaged in a dialogue. An audio-visual spatiotemporal diarization model is proposed. The model is well suited for challenging scenarios that consist of several participants engaged in multi-party interaction while they move around and turn their heads towards the other participants rather than facing the cameras and the microphones. Multiple-person visual tracking is combined with multiple speech-source localization in order to tackle the speech-to-person association problem. The latter is solved within a novel audio-visual fusion method on the following grounds: binaural spectral features are first extracted from a microphone pair, then a supervised audio-visual alignment technique maps these features onto an image, and finally a semi-supervised clustering method assigns binaural spectral features to visible persons. The main advantage of this method over previous work is that it processes in a principled way speech signals uttered simultaneously by multiple persons. The diarization itself is cast into a latent-variable temporal graphical model that infers speaker identities and speech turns, based on the output of an audio-visual association process, executed at each time slice, and on the dynamics of the diarization variable itself. The proposed formulation yields an efficient exact inference procedure. A novel dataset, that contains audio-visual training data as well as a number of scenarios involving several participants engaged in formal and informal dialogue, is introduced. The proposed method is thoroughly tested and benchmarked with respect to several state-of-the art diarization algorithms.

Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

AudioVSR: Enhancing Video Speech Recognition with Audio Data

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

A Study on Joint Modeling and Data Augmentation of Multi-Modalities for Audio-Visual Scene Classification

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Late Audio-Visual Fusion for In-The-Wild Speaker Diarization

Uncertainty-Guided End-to-End Audio-Visual Speaker Diarization for Far-Field Recordings

Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition

Joint Training of Speaker Embedding Extractor, Speech and Overlap Detection for Diarization

Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization

Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion

The FlySpeech Audio-Visual Speaker Diarization System for MISP Challenge 2022

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module

Visual-Enhanced End-to-End Neural Diarization

Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

Robust Audio-Visual ASR with Unified Cross-Modal Attention

Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy