PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents

Qisen Yang,Zekun Wang,Honghui Chen,Shenzhi Wang,Yifan Pu,Xin Gao,Wenhao Huang,Shiji Song,Gao Huang

2024-08-29

Abstract:Psychological measurement is essential for mental health, self-understanding, and personal development. Traditional methods, such as self-report scales and psychologist interviews, often face challenges with engagement and accessibility. While game-based and LLM-based tools have been explored to improve user interest and automate assessment, they struggle to balance engagement with generalizability. In this work, we propose PsychoGAT (Psychological Game AgenTs) to achieve a generic gamification of psychological assessment. The main insight is that powerful LLMs can function both as adept psychologists and innovative game designers. By incorporating LLM agents into designated roles and carefully managing their interactions, PsychoGAT can transform any standardized scales into personalized and engaging interactive fiction games. To validate the proposed method, we conduct psychometric evaluations to assess its effectiveness and employ human evaluators to examine the generated content across various psychological constructs, including depression, cognitive distortions, and personality traits. Results demonstrate that PsychoGAT serves as an effective assessment tool, achieving statistically significant excellence in psychometric metrics such as reliability, convergent validity, and discriminant validity. Moreover, human evaluations confirm PsychoGAT's enhancements in content coherence, interactivity, interest, immersion, and satisfaction.

Computation and Language,Computers and Society,Human-Computer Interaction,Machine Learning,Multiagent Systems

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address several key issues in psychometrics:

1. **Lack of Engagement**: Traditional psychometric methods, such as self-report scales and psychologist interviews, are often monotonous, especially among younger groups, which may lead to participant resistance.
2. **Poor Accessibility**: Expert interviews require a significant amount of human resources, and these resources are scarce, making psychological assessments difficult to reach those in need.
3. **Balancing Automation and Generalization**: Although game-based and large language model (LLM)-based psychological assessment tools have been explored, they struggle to maintain broad applicability and generalization while increasing user interest.

To address these issues, the paper proposes a new psychometric paradigm—PsychoGAT (Psychological Game Agent). PsychoGAT enhances engagement and assessment effectiveness by transforming standardized psychometric scales into personalized interactive novel games, leveraging powerful LLMs as skilled psychologists and innovative game designers. Specifically, PsychoGAT achieves this goal through the following means:

- **Multi-Agent Framework**: Includes three main agents—game designer, game controller, and critic—responsible for creating game settings, generating interactive content, and optimizing user experience, respectively.
- **Personalization and Interactivity**: Allows players to choose game types and themes, making the assessment process more enjoyable and immersive.
- **Psychometric Validation**: Validates the effectiveness and user experience of PsychoGAT through psychometric assessments and human evaluations.

Experimental results indicate that PsychoGAT performs excellently on psychometric indicators (such as reliability, convergent validity, and discriminant validity) and significantly improves user experience-related metrics (such as coherence, interactivity, interest, immersion, and satisfaction).