Abstract:Speaker diarization, the process of segmenting an audio stream or transcribed speech content into homogenous partitions based on speaker identity, plays a crucial role in the interpretation and analysis of human speech. Most existing speaker diarization systems rely exclusively on unimodal acoustic information, making the task particularly challenging due to the innate ambiguities of audio signals. Recent studies have made tremendous efforts towards audio-visual or audio-semantic modeling to enhance performance. However, even the incorporation of up to two modalities often falls short in addressing the complexities of spontaneous and unstructured conversations. To exploit more meaningful dialogue patterns, we propose a novel multimodal approach that jointly utilizes audio, visual, and semantic cues to enhance speaker diarization. Our method elegantly formulates the multimodal modeling as a constrained optimization problem. First, we build insights into the visual connections among active speakers and the semantic interactions within spoken content, thereby establishing abundant pairwise constraints. Then we introduce a joint pairwise constraint propagation algorithm to cluster speakers based on these visual and semantic constraints. This integration effectively leverages the complementary strengths of different modalities, refining the affinity estimation between individual speaker embeddings. Extensive experiments conducted on multiple multimodal datasets demonstrate that our approach consistently outperforms state-of-the-art speaker diarization methods.

An Analysis of Speaker Diarization Fusion Methods for the First DIHARD Challenge

Speaker Diarization with Enhancing Speech for the First DIHARD Challenge

Scenario-Dependent Speaker Diarization for DIHARD-III Challenge

DIHARD II is Still Hard: Experimental Results and Discussions from the DKU-LENOVO Team

A Deep Analysis of Speech Separation Guided Diarization Under Realistic Conditions

Exploring Detection-based Method For Speaker Diarization @ Ego4D Audio-only Diarization Challenge 2022

Late Audio-Visual Fusion for In-The-Wild Speaker Diarization

The research of speaker diarization based on BIC and G_PLDA

BUT System Description for DIHARD Speech Diarization Challenge 2019

A Quick and Effective Speaker Diarization System.

The Third DIHARD Diarization Challenge

Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization

The RoyalFlush Automatic Speech Diarization and Recognition System for In-Car Multi-Channel Automatic Speech Recognition Challenge

Royalflush Speaker Diarization System for ICASSP 2022 Multi-channel Multi-party Meeting Transcription Challenge

DCF-DS: Deep Cascade Fusion of Diarization and Separation for Speech Recognition under Realistic Single-Channel Conditions

Progressive Multi-Target Network Based Speech Enhancement with Snr-Preselection for Robust Speaker Diarization

Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization

Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020

Uncertainty-Guided End-to-End Audio-Visual Speaker Diarization for Far-Field Recordings

The DKU-MSXF Diarization System for the VoxCeleb Speaker Recognition Challenge 2023

Speaker Diarization with Lexical Information