Abstract:This paper presents a multimodal dialogue system with user personality adaptation and multiple nonverbal behavior generation for a conversational android robot that plays a customer service role in recommending travel plans. It is useful to estimate the user's personality to realize a dialogue system that adapts the dialogue strategy to individual users. To improve the user's impression of the conversation robot as a customer service agent, we must control the nonverbal behavior of the robot, such as its facial expressions and motions. Moreover, we need to design the dialogue strategy to provide information and make users enjoy the conversation by adding an ice-breaking step. Against this background, we develop a dialogue system prototype that adjusts the dialogue strategy based on the user's personality, as estimated by a pre-trained multimodal sensing model. Furthermore, we implement appropriate nonverbal behavior patterns for each situation, including the voice, motion, and facial expressions of the android robot, to provide it appropriate and polite service agent behaviors. A personality assessment game is introduced into the dialogue to prevent the user from becoming bored. The game plays an ice-breaking role in the dialogue. To let the user engage with the robot, we implement a complement to the user as a dialogue act. We evaluate the proposed multimodal dialogue system for customer service by means of participation in a dialogue robot competition. The fair evaluation results from 26 dialogue users who conversed with multiple systems show that the proposed dialogue system achieved the best score among 13 teams participating in the preliminary round of the competition, indicating that the user personality adaptation and various elements implemented in our dialogue system improve the dialogue experience of general users. The results of a third-party evaluation and multiple regression analysis show that natural response is the most important factor contributing to user impression.

User-Personality Classification Based on the Non-Verbal Cues from Spoken Conversations

Identification of Speech Characteristics to Distinguish Human Personality of Introversive and Extroversive Male Groups

Confiding in and Listening to Virtual Agents: The Effect of Personality

A Personality-Based Instant Messaging Software User Classification Model

Speech Personality Recognition Based on Annotation Classification Using Log-Likelihood Distance and Extraction of Essential Audio Features.

Personality prediction from task-oriented and open-domain human–machine dialogues

PersoNet: A Novel Framework for Personality Classification-Based Apt Customer Service Agent Selection

A Comprehensive Multimodal Humanoid System for Personality Assessment Based on the Big Five Model

Feasibility of Big Data Analytics to Assess Personality Based on Voice Analysis

Personality Style Recognition via Machine Learning: Identifying Anaclitic and Introjective Personality Styles from Patients' Speech

Is ChatGPT a Good Personality Recognizer? A Preliminary Study

Personality-adapted multimodal dialogue system

PersonalityScanner: Exploring the Validity of Personality Assessment Based on Multimodal Signals in Virtual Reality

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning

A multimodal dialogue system for customer service based on user personality adaptation and dialogue strategies

Assessing the Impact of Chatbot-Human Personality Congruence on User Behavior: A Chatbot-Based Advising System Case

Rapport with Virtual Agents: What Do Human Social Cues and Personality Explain?

Comparing approaches for mitigating intergroup variability in personality recognition

Automatic Text-based Personality Recognition on Monologues and Multiparty Dialogues Using Attentive Networks and Contextual Embeddings

Feature Learning from Spectrograms for Assessment of Personality Traits