Abstract:Speaker diarization consists of assigning speech signals to people engaged in a dialogue. An audio-visual spatiotemporal diarization model is proposed. The model is well suited for challenging scenarios that consist of several participants engaged in multi-party interaction while they move around and turn their heads towards the other participants rather than facing the cameras and the microphones. Multiple-person visual tracking is combined with multiple speech-source localization in order to tackle the speech-to-person association problem. The latter is solved within a novel audio-visual fusion method on the following grounds: binaural spectral features are first extracted from a microphone pair, then a supervised audio-visual alignment technique maps these features onto an image, and finally a semi-supervised clustering method assigns binaural spectral features to visible persons. The main advantage of this method over previous work is that it processes in a principled way speech signals uttered simultaneously by multiple persons. The diarization itself is cast into a latent-variable temporal graphical model that infers speaker identities and speech turns, based on the output of an audio-visual association process, executed at each time slice, and on the dynamics of the diarization variable itself. The proposed formulation yields an efficient exact inference procedure. A novel dataset, that contains audio-visual training data as well as a number of scenarios involving several participants engaged in formal and informal dialogue, is introduced. The proposed method is thoroughly tested and benchmarked with respect to several state-of-the art diarization algorithms.

BUT System Description for DIHARD Speech Diarization Challenge 2019

Speaker Diarization with Enhancing Speech for the First DIHARD Challenge

DIHARD II is Still Hard: Experimental Results and Discussions from the DKU-LENOVO Team

TalTech-IRIT-LIS Speaker and Language Diarization Systems for DISPLACE 2024

An Analysis of Speaker Diarization Fusion Methods for the First DIHARD Challenge

The Second DIHARD Diarization Challenge: Dataset, task, and baselines.

Scenario-Dependent Speaker Diarization for DIHARD-III Challenge

The Third DIHARD Diarization Challenge

Summary of the DISPLACE Challenge 2023 - DIarization of SPeaker and LAnguage in Conversational Environments

Ivec-PLDA-AHC Priors for VB-HMM Speaker Diarization System

A Quick and Effective Speaker Diarization System.

Bayesian HMM Based x-Vector Clustering for Speaker Diarization

Discriminative Training of VBx Diarization

The FlySpeech Audio-Visual Speaker Diarization System for MISP Challenge 2022

Joint Training of Speaker Embedding Extractor, Speech and Overlap Detection for Diarization

GIST-AiTeR System for the Diarization Task of the 2022 VoxCeleb Speaker Recognition Challenge

TCG CREST System Description for the Second DISPLACE Challenge

Exploring Detection-based Method For Speaker Diarization @ Ego4D Audio-only Diarization Challenge 2022

The DKU-MSXF Diarization System for the VoxCeleb Speaker Recognition Challenge 2023

Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion

The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognition