Abstract:Embodied agents designed to assist users with tasks must engage in natural language interactions, interpret instructions, execute actions, and communicate effectively to resolve issues. However, collecting large-scale, diverse datasets of situated human-robot dialogues to train and evaluate such agents is expensive, labor-intensive, and time-consuming. To address this challenge, we propose building a large language model (LLM)-based user agent that can simulate user behavior during interactions with an embodied agent in a virtual environment. Given a user goal (e.g., make breakfast), at each time step, the user agent may observe" the robot actions or speak" to either intervene with the robot or answer questions. Such a user agent assists in improving the scalability and efficiency of embodied dialogues dataset generation and is critical for enhancing and evaluating the robot's interaction and task completion ability, as well as for research in reinforcement learning using AI feedback. We evaluate our user agent's ability to generate human-like behaviors by comparing its simulated dialogues with the TEACh dataset. We perform three experiments: zero-shot prompting to predict dialogue acts, few-shot prompting, and fine-tuning on the TEACh training subset. Results show the LLM-based user agent achieves an F-measure of 42% with zero-shot prompting and 43.4% with few-shot prompting in mimicking human speaking behavior. Through fine-tuning, performance in deciding when to speak remained stable, while deciding what to say improved from 51.1% to 62.5%. These findings showcase the feasibility of the proposed approach for assessing and enhancing the effectiveness of robot task completion through natural language communication.

Measuring Conversational Fluidity in Automated Dialogue Agents

On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems

Towards Automatic Evaluation of Task-Oriented Dialogue Flows

Psychological Metrics for Dialog System Evaluation

Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

On Evaluating and Comparing Open Domain Dialog Systems

FFAEval: Evaluating Dialogue System Via Free-For-All Ranking

Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems

Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs

PCDialogEval: Persona and Context Aware Emotional Dialogue Evaluation

FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

Simulating User Agents for Embodied Conversational-AI

Leveraging Implicit Feedback from Deployment Data in Dialogue

Actionable conversational quality indicators for improving task-oriented dialog systems

Synthetic Dialogue Dataset Generation using LLM Agents

A Measure for Dialog Complexity and its Application in Streamlining Service Operations