Abstract:It enhances user experience by making the talking avatar be sensitive to user behaviors in human computer interaction (HCI). In this study, we combine user's multi-modal behaviors with behaviors' historical information in dialog management (DM) to improve the avatar's sensitivity not only to user explicit behavior (speech command) but also to user supporting expression (emotion and gesture, etc.). In the dialog management, according to the different contributions of facial expression, gesture and head motion to speech comprehension, we divide the user's multi-modal behaviors into three categories: complementation, conflict and independence. The behavior categories could be first automatically obtained from a short-term and time-dynamic (STTD) fusion model with audio-visual input. Different behavior category leads to different avatar's response in later dialog turns. Usually, the conflict behavior reflects user's ambiguous intention (for example: user says "no" while he (her) is smiling). In this case, the trial-and-error schema is adopted to eliminate the conversation ambiguity. For the later dialog process, we divide all the avatar dialog states into four types: "Ask", "Answer", "Chat" and "Forget". With the detection of complementation and independence behaviors, the user supporting expression as well as his (her) explicit behavior could be estimated as triggers for topic maintenance or transfer among four dialog states. At the first section of experiments, we discuss the reliability of STTD model for user behavior classification. Based on the proposed dialog management and STTD model, we continue to construct a drive route information query system by connecting the user behavior sensitive dialog management (BSDM) to a 3D talking avatar. The practical conversation records of avatar with different users show that the BSDM makes the avatar be able to understand and be sensitive to the users' facial expressions, emotional voice and gesture, which improves user experience on multi-modal human computer conversation.

MDS: a multimodal-based dialog system

Handtalker: A Multimodal Dialog System Using Sign Language and 3-D Virtual Human

Hybrid Silent Speech Interface Through Fusion of Electroencephalography and Electromyography

Building Language Communication between Deaf People and Hearing Society through Multimodal Human-Computer Interface

A New Mmwave-Speech Multimodal Speech System for Voice User Interface

Magic Music Desk: A Tangible and Ubiquitous Multimodal Music and Speech System

A Multimodal Emotional Communication Based Humans-Robots Interaction System

CSLDS: Chinese Sign Language Dialog System

A Multimodal Approach of Generating 3D Human-Like Talking Agent.

User Behavior Fusion in Dialog Management with Multi-Modal History Cues

HandTalker II: a Chinese sign language recognition and synthesis system

Magic music desk: a multi-modal embodied interactive desk

Text to Avatar in Multi-modal Human Computer Interface

Simple Dialogue System with AUDITED

A multimodal human-robot sign language interaction framework applied in social robots

Designing A Distance Learning System for English Reading by Applying Multimodal Discourse Analysis Theory

User Attention-guided Multimodal Dialog Systems

Human-Computer Chinese Sign Language Interaction System

A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

M$^{3}$V: A multi-modal multi-view approach for Device-Directed Speech Detection

A Multimodal Assistive System for Helping Visually Impaired in Social Interactions