Abstract:This work investigates how personality expression and embodiment affect personality perception and learning in educational conversational agents. We extend an existing personality-driven conversational agent framework by integrating LLM-based conversation support tailored to an educational application. We describe a user study built on this system to evaluate two distinct personality styles: high extroversion and agreeableness and low extroversion and agreeableness. For each personality style, we assess three models: (1) a dialogue-only model that conveys personality through dialogue, (2) an animated human model that expresses personality solely through dialogue, and (3) an animated human model that expresses personality through both dialogue and body and facial animations. The results indicate that all models are positively perceived regarding both personality and learning outcomes. Models with high personality traits are perceived as more engaging than those with low personality traits. We provide a comprehensive quantitative and qualitative analysis of perceived personality traits, learning parameters, and user experiences based on participant ratings of the model types and personality styles, as well as users' responses to open-ended questions.
What problem does this paper attempt to address?
### The Problem the Paper Attempts to Solve
This paper aims to explore how the embodiment and personality expression of virtual educational agents, when expressing specific personality traits, affect learning outcomes and personality perception. Specifically, the researchers extended an existing personality-driven conversational agent framework by integrating a large language model (LLM)-supported dialogue system and evaluated the impact of different personality styles (high extraversion and agreeableness vs. low extraversion and agreeableness) and different model types (dialogue-only, animated human model expressing personality through dialogue only, animated human model expressing personality through dialogue and body and facial animations) on learning outcomes and personality perception.
### Research Background
Virtual humans have great potential in educational activities, providing on-demand personalized learning experiences. They not only convey information but also establish social connections, build rapport, and motivate users. With the development of large language models (LLMs), virtual humans can more effectively assume various roles and personality traits, providing information on any topic. This technological advancement enables virtual humans to offer customized experiences for learners with different preferences and needs.
### Research Questions
1. **Impact of Personality Style on Perceived Personality** (RQ1): Does different personality styles affect users' perception of the virtual agent's personality?
2. **Impact of Model Type on Perceived Personality** (RQ2): Does different model types affect users' perception of the virtual agent's personality?
3. **Impact of Model Type on Learning Outcomes** (RQ3): Does different model types affect learning outcomes?
4. **Correlation Between Learning Outcomes and Personality Perception** (RQ4): Is there a correlation between learning outcomes and personality perception?
### Hypotheses
- **Hypothesis 1** (H1): The learning outcome scores of embodied agents will be higher than those of dialogue-only agents, reflected in higher learning scores, quality scores, and engagement scores.
- **Hypothesis 2** (H2): Agents expressing high extraversion and agreeableness will score higher in learning outcomes than agents expressing low extraversion and agreeableness, also reflected in higher learning scores, quality scores, and engagement scores.
### Research Methods
The researchers designed a user experiment comparing the performance of three models (dialogue-only model, animated human model expressing personality through dialogue only, animated human model expressing personality through dialogue and body and facial animations) under two personality styles (high extraversion and agreeableness vs. low extraversion and agreeableness). Participants were randomly assigned to different combinations of models and personality styles, interacted with the virtual agents, completed a series of learning tasks, and filled out questionnaires to evaluate learning outcomes, quality, engagement, and perception of the agent's personality.
### Data Analysis
The researchers used both quantitative and qualitative methods to analyze the data. Quantitative analysis included using a 5-point Likert scale to assess learning outcomes, quality, and engagement, and using the short version of the Big Five Inventory (BFI-2-XS) to evaluate the agent's personality traits. Qualitative analysis involved in-depth thematic analysis of responses to open-ended questions, identifying common themes and individual differences among participants.
### Conclusion
The research results indicate that all models received positive evaluations in terms of personality perception and learning outcomes. Models with high personality traits were considered more attractive. The researchers also provided comprehensive quantitative and qualitative analyses, detailing participants' ratings and feedback on different models and personality styles.
Through this study, the authors hope to provide valuable insights for the design and development of virtual educational agents, particularly in terms of personality expression and embodiment.