Addressee Detection Using Facial and Audio Features in Mixed Human–Human and Human–Robot Settings: A Deep Learning Framework

Fiseha B. Tesema,Jason Gu,Wei Song,Hong Wu,Shiqiang Zhu,Zheyuan Lin,Min Huang,Wen Wang,Rajesh Kumar

DOI: https://doi.org/10.1109/msmc.2022.3224843

2023-01-01

IEEE Systems Man and Cybernetics Magazine

Abstract:Addressee detection (AD) enables robots to interact smoothly with a human by distinguishing whether it is being addressed. However, this has not been widely explored. The few studies that have explored this area focused on a human-to-human or human-to-robot conversation confined inside a meeting room using gaze and utterance. These works used statistical and rule-based approaches, which tend to depend on specific settings. Further, they did not fully leverage the available audio and visual information or the short-term and long-term segments, and they have not explored combining important conversation cues—the facial and audio features. In addition, no audiovisual spatiotemporal annotated dataset captured in mixed human-to-human and human-to-robot settings is available to support exploring the area using new approaches.