Abstract:The recent success of large language models (LLMs) has attracted widespread interest to develop role-playing conversational agents personalized to the characteristics and styles of different speakers to enhance their abilities to perform both general and special purpose dialogue tasks. However, the ability to personalize the generated utterances to speakers, whether conducted by human or LLM, has not been well studied. To bridge this gap, our study introduces a novel evaluation challenge: speaker verification in agent-generated conversations, which aimed to verify whether two sets of utterances originate from the same speaker. To this end, we assemble a large dataset collection encompassing thousands of speakers and their utterances. We also develop and evaluate speaker verification models under experiment setups. We further utilize the speaker verification models to evaluate the personalization abilities of LLM-based role-playing models. Comprehensive experiments suggest that the current role-playing models fail in accurately mimicking speakers, primarily due to their inherent linguistic characteristics.

What problem does this paper attempt to address?

The main aim of this paper is to address the following issues: 1. **Defining and Evaluating the Personalization Ability of Role-Playing Dialogue Agents**: With the development of large language models (LLMs), role-playing dialogue agents can simulate speakers with different personal attributes and language styles to enhance their ability to perform general and specific dialogue tasks. However, how to evaluate whether the dialogues generated by these agents are truly personalized to the characteristics of the target speakers has not been fully studied. 2. **Introducing the Speaker Verification Task**: To fill this research gap, the authors introduce a new evaluation challenge—speaker verification in agent-generated dialogues. The goal of this task is to verify whether two sets of dialogues come from the same speaker, thereby measuring whether the dialogue agent can accurately mimic the language style and personal characteristics of the target speaker. 3. **Developing a Speaker Verification Dataset and Model**: The authors constructed a large-scale dataset containing thousands of speakers and their dialogues and developed speaker verification models for experimental evaluation. Through these models, the personalization ability of LLM-based role-playing models was further assessed. 4. **Revealing the Limitations of Existing Role-Playing Models**: Comprehensive experimental results indicate that current role-playing models have difficulty accurately mimicking speakers, especially in retaining the unique language style and personal traits of the speakers. 5. **Proposing Evaluation Metrics and Framework**: To better evaluate the performance of role-playing models, the paper proposes two metrics—Simulation Score and Distinction Score—to assess the consistency of agent-generated dialogues with the target speaker's actual dialogues and the distinction between different speakers. In summary, this study aims to evaluate and improve the personalization level of role-playing dialogue agents by introducing the speaker verification task and developing corresponding models.

Speaker Verification in Agent-Generated Conversations

VarASV: Enabling Pitch-variable Automatic Speaker Verification Via Multi-task Learning

Character-LLM: A Trainable Agent for Role-Playing

Enabling Conversational Interaction with Mobile UI using Large Language Models

Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

GameEval: Evaluating LLMs on Conversational Games

User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants

Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

Simulating User Agents for Embodied Conversational-AI

Role-Play Zero-Shot Prompting with Large Language Models for Open-Domain Human-Machine Conversation

LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models

Large Language Model based Situational Dialogues for Second Language Learning

LLM Roleplay: Simulating Human-Chatbot Interaction

RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Models

Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk

Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data

Leveraging LLMs for Dialogue Quality Measurement

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games