Abstract:It enhances user experience by making the talking avatar be sensitive to user behaviors in human computer interaction (HCI). In this study, we combine user's multi-modal behaviors with behaviors' historical information in dialog management (DM) to improve the avatar's sensitivity not only to user explicit behavior (speech command) but also to user supporting expression (emotion and gesture, etc.). In the dialog management, according to the different contributions of facial expression, gesture and head motion to speech comprehension, we divide the user's multi-modal behaviors into three categories: complementation, conflict and independence. The behavior categories could be first automatically obtained from a short-term and time-dynamic (STTD) fusion model with audio-visual input. Different behavior category leads to different avatar's response in later dialog turns. Usually, the conflict behavior reflects user's ambiguous intention (for example: user says "no" while he (her) is smiling). In this case, the trial-and-error schema is adopted to eliminate the conversation ambiguity. For the later dialog process, we divide all the avatar dialog states into four types: "Ask", "Answer", "Chat" and "Forget". With the detection of complementation and independence behaviors, the user supporting expression as well as his (her) explicit behavior could be estimated as triggers for topic maintenance or transfer among four dialog states. At the first section of experiments, we discuss the reliability of STTD model for user behavior classification. Based on the proposed dialog management and STTD model, we continue to construct a drive route information query system by connecting the user behavior sensitive dialog management (BSDM) to a 3D talking avatar. The practical conversation records of avatar with different users show that the BSDM makes the avatar be able to understand and be sensitive to the users' facial expressions, emotional voice and gesture, which improves user experience on multi-modal human computer conversation.

Predicting and Tagging Dialog-Act Using MDP and SVM

Applying Support Vector Machines to Voice Activity Detection

Guiding Attention in Sequence-to-Sequence Models for Dialogue Act Prediction

Identifying Discourse Markers in Spoken Dialog

Memory-Augmented Dialogue Management for Task-Oriented Dialogue Systems

Act-Aware Slot-Value Predicting in Multi-Domain Dialogue State Tracking

Implementing Chinese new word discovery and POS tagging based on support vector machine

Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech

DialogAct2Vec: Towards End-to-End Dialogue Agent by Multi-Task Representation Learning

A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognition

STN4DST: A Scalable Dialogue State Tracking based on Slot Tagging Navigation

"Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs

Improved Dynamic Memory Network for Dialogue Act Classification with Adversarial Training

Multimodal Dialogue Response Generation Based on Selective Attention and Gating Mechanisms

Interactive Spoken Content Retrieval with Different Types of Actions Optimized by a Markov Decision Process

Modelling User Behaviour In The His-Pomdp Dialogue Manager

A Post-Processing Method for Detecting Unknown Intent of Dialogue System Via Pre-Trained Deep Neural Network Classifier

Unknown Intent Detection Method Based on Model Post-Processing

Interactive spoken content retrieval by extended query model and continuous state space Markov Decision Process

TIER: A Novel Model Based on POS Information to Generate Dialogue.

User Behavior Fusion in Dialog Management with Multi-Modal History Cues