Abstract:With the rapid development of multimedia technology, personnel verification systems have become increasingly important in the security field and identity verification. However, unimodal verification systems have performance bottlenecks in complex scenarios, thus triggering the need for multimodal feature fusion methods. The main problem with audio-visual multimodal feature fusion is how to effectively integrate information from different modalities to improve the accuracy and robustness of the system for individual identity. In this paper, we focus on how to improve multimodal person verification systems and how to combine audio and visual features. In this study, we use pretrained models to extract the embeddings from each modality and then perform fusion model experiments based on these embeddings. The baseline approach in this paper involves taking the fusion feature and passing it through a fully connected (FC) layer. Building upon this baseline, we propose three fusion models based on attentional mechanisms: attention, gated, and inter-attention. These fusion models are trained on the VoxCeleb1 development set and tested on the evaluation sets of the VoxCeleb1, NIST SRE19, and CNC-AV datasets. On the VoxCeleb1 dataset, the best system performance achieved in this study was an equal error rate (EER) of 0.23% and a detection cost function (minDCF) of 0.011. On the evaluation set of NIST SRE19, the EER was 2.60% and the minDCF was 0.283. On the evaluation set of the CNC-AV set, the EER was 11.30% and the minDCF was 0.443. These experimental results strongly demonstrate that the proposed fusion method can significantly improve the performance of multimodal character verification systems.

Weight Estimation for Audio-Visual Multi-level Fusion in Bimodal Speaker Identification

Multi-level Fusion of Audio and Visual Features for Speaker Identification

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

Dynamic bayesian networks for audio-visual speaker recognition

Bimodal speaker identification using dynamic bayesian network

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain

Robust Audio-Visual Speech Recognition Based on Hybrid Fusion

Integration of audio-visual information for multi-speaker multimedia speaker recognition

Improving Visual Speech Enhancement Network by Learning Audio-visual Affinity with Multi-head Attention

Audio-Visual Information Fusion Using Cross-Modal Teacher-Student Learning for Voice Activity Detection in Realistic Environments

Audio-Visual Speaker Verification via Joint Cross-Attention

Modeling the Synchrony between Audio and Visual Modalities for Speaker Identification

Exploiting Prosodic Information for Speaker Recognition

Attentive Feature Fusion for Robust Speaker Verification

Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition

Deep Multimodal Learning for Audio-Visual Speech Recognition

Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Audio-Visual Fusion Based on Interactive Attention for Person Verification

Multimodal Fusion Method with Spatiotemporal Sequences and Relationship Learning for Valence-Arousal Estimation