Abstract:Active speaker detection (ASD) refers to detecting the speaking person among visible human instances in a video. Existing methods widely employed a similar audiovisual fusion approach, the concatenation. Although such a fusion approach is often argued to help enhance performance, it must be noted that neither feature modalities play an equal role. It forces the backend network to focus on learning intramodal rather than intermodal features. Another concern is that since the concatenation doubles the fused feature dimension that feeds from the audio and video module, it creates a higher computational overhead for the backend network. To address these problems, this work hypothesizes that instead of leveraging deterministic fusion operation, employing an efficient fusion technique may assist the network in learning efficiently and improve detection accuracy. This work proposes an efficient audiovisual fusion (AVF) with fewer feature dimensions that captures the correlations between facial regions and sound signals, focusing more on the discriminative facial features and associating them with the corresponding audio features. Furthermore, previous ASD works focus only on improving ASD performance by creating a large computational overhead using complex techniques such as adding sophisticated postprocessing, applying smoothing techniques on the classifier to refine the network outputs at multiple stages, or assembling the multiple network outputs. This work proposed a simple yet effective end-to-end ASD using the newly proposed feature fusion approach, the AVF. The proposed framework attained a mAP of 84.384% on the validation set of the most challenging audiovisual speaker detection benchmark, the AVA-ActiveSpeaker. With this, this work outperformed previous works that did not apply the postprocessing tasks and attained competitive detection accuracy compared to other works that employed different postprocessing tasks. The proposed model also learns better on the unsynchronized raw AVA-ActiveSpeaker dataset. The ablation experiments under different image scale settings and noisy signals show the AFV’s effectiveness and robustness than the concatenation operation.

Robust Active Speaker Detection in Noisy Environments

Efficient Audiovisual Fusion for Active Speaker Detection.

Real-time Architecture for Audio-Visual Active Speaker Detection.

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

Target Active Speaker Detection with Audio-visual Cues

An Efficient and Streaming Audio Visual Active Speaker Detection System

A Light Weight Model for Active Speaker Detection

AS-Net: active speaker detection using deep audio-visual attention

Enhancing Real-World Active Speaker Detection with Multi-Modal Extraction Pre-Training

ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection

Robust Audio Sensing with Multi-Sound Classification.

Best of Both Worlds: Multi-Task Audio-Visual Automatic Speech Recognition and Active Speaker Detection

Visually Supervised Speaker Detection and Localization via Microphone Array

Rethinking Audio-visual Synchronization for Active Speaker Detection

Robust Prototype Learning for Anomalous Sound Detection.

Active Speaker Detection Using Audio, Visual, and Depth Modalities: A Survey

Improving Anomalous Sound Detection via Low-Rank Adaptation Fine-Tuning of Pre-Trained Audio Models

E-Talk: Accelerating Active Speaker Detection with Audio-Visual Fusion and Edge-Cloud Computing

How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild

Robust Speaker Identification In Noise Using Missing Data Technique And Auditory Masking

Active Speaker Detection in Fisheye Meeting Scenes with Scene Spatial Spectrums