Abstract:It enhances user experience by making the talking avatar be sensitive to user behaviors in human computer interaction (HCI). In this study, we combine user's multi-modal behaviors with behaviors' historical information in dialog management (DM) to improve the avatar's sensitivity not only to user explicit behavior (speech command) but also to user supporting expression (emotion and gesture, etc.). In the dialog management, according to the different contributions of facial expression, gesture and head motion to speech comprehension, we divide the user's multi-modal behaviors into three categories: complementation, conflict and independence. The behavior categories could be first automatically obtained from a short-term and time-dynamic (STTD) fusion model with audio-visual input. Different behavior category leads to different avatar's response in later dialog turns. Usually, the conflict behavior reflects user's ambiguous intention (for example: user says "no" while he (her) is smiling). In this case, the trial-and-error schema is adopted to eliminate the conversation ambiguity. For the later dialog process, we divide all the avatar dialog states into four types: "Ask", "Answer", "Chat" and "Forget". With the detection of complementation and independence behaviors, the user supporting expression as well as his (her) explicit behavior could be estimated as triggers for topic maintenance or transfer among four dialog states. At the first section of experiments, we discuss the reliability of STTD model for user behavior classification. Based on the proposed dialog management and STTD model, we continue to construct a drive route information query system by connecting the user behavior sensitive dialog management (BSDM) to a 3D talking avatar. The practical conversation records of avatar with different users show that the BSDM makes the avatar be able to understand and be sensitive to the users' facial expressions, emotional voice and gesture, which improves user experience on multi-modal human computer conversation.

Spatial-Temporal Aligned Multi-Agent Learning for Visual Dialog Systems

S2rl

Multi-goal Multi-Agent Learning for Task-Oriented Dialogue with Bidirectional Teacher-Student Learning

Structure-Aware Multimodal Sequential Learning for Visual Dialog

SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems

Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learning

DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention

Learning Efficient Multi-Agent Cooperative Visual Exploration

Multi-Speaker Video Dialog with Frame-Level Temporal Localization

InsightSee: Advancing Multi-agent Vision-Language Models for Enhanced Visual Understanding

Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

DialogAct2Vec: Towards End-to-End Dialogue Agent by Multi-Task Representation Learning

Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition

Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality

Joint-Predictive Representations for Multi-Agent Reinforcement Learning

Integrating User and Agent Models: A Deep Task-Oriented Dialogue System

Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI

User Behavior Fusion in Dialog Management with Multi-Modal History Cues

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Multi-agent Planning using Visual Language Models

Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction