Abstract:Recently, multi-speaker multimedia speaker recognition (MMSR) has garnered significant attention. Although prior research primarily focused on the back-end score level fusion of audio and visual information, this study delves into innovative techniques for integrating audio and visual cues from the front-end representations of both speaker's voice and face. The first method introduces the use of visual information to estimate the number of speakers. This solution addresses the challenges of estimating speaker numbers in multi-speaker conversations, especially in noisy environments. Subsequently, agglomerative hierarchical clustering is employed for speaker diarization, proving beneficial for MMSR. This approach is termed video aiding audio fusion (VAAF). The second method innovates by introducing a ratio factor to create a multimedia vector (M-vector) which concatenates face embeddings with x-vector. This amalgamation encapsulates both audio and visual cues. The resulting M-vector is then leveraged for MMSR. We name this method as video interacting audio fusion (VIAF). Experimental results on the NIST SRE 2019 audio-visual corpus reveal that the VAAF-based MMSR achieves a 6.94% and 8.31% relative reduction in minDCF and actDCF, respectively, when benchmarked against zero-effort systems. Additionally, the VIAF-based MMSR realizes a 12.08% and 12.99% relative reduction in minDCF and actDCF, respectively, compared to systems that solely utilize face embeddings. Notably, when combining both methods, the minDCF and actDCF metrics are further optimized, reaching 0.098 and 0.102, respectively.

Audio-Visual System for Robust Speaker Recognition.

Speaker Recognition Technology Based on Lip Movement

Speaker Identification System Based on Lip-Motion Feature.

A Speaker Identification System for Video Content Analysis

Speaker Recognition Based on Lip-reading: an Overview

Lip motion recognition of speaker based on SIFT

Visual-Audio Integration For User Authentication System Of Partner Robots

Integration of audio-visual information for multi-speaker multimedia speaker recognition

Visual Speaker Authentication By A Cnn-Based Scheme With Discriminative Segment Analysis

A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model

Speaker verification system based on M-vector and support vector machine

Combined GMM-UBM and SVM Speaker Identification System

Audio-visual Human Recognition Using Semi-Supervised Spectral Learning and Hidden Markov Models

A Probabilistic Principal Component Analysis Based Hidden Markov Model For Audio-Visual Speech Recognition

Audio-visual Speech Recognition Integrating 3D Lip Information Obtained from the Kinect

The Visual-Audio Integrated Recognition Method for User Authentication System of Partner Robots.

An Audio-Visual Speech Recognition Framework Based on Articulatory Features.

Study to Speaker Recognition Using RVM

SPEAKER RETRIEVAL FOR TV SHOW VIDEOS BY ASSOCIATING AUDIO SPEAKER RECOGNITION RESULT TO VISUAL FACES

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

3D Convolutional Neural Networks Based Speaker Identification and Authentication.