Abstract:Recent advancements in natural language and Large Language Models (LLMs) have enabled AI agents to simulate human-like interactions within virtual worlds. However, these interactions still face limitations in complexity and flexibility, particularly in scenarios involving multiple characters and novel objects. Pre-defining all interactable objects in the agent's world model presents challenges, and conveying implicit intentions to multiple characters through complex interactions remains difficult. To address these issues, we propose integrating virtual Game Masters (GMs) into the agent's world model, drawing inspiration from Tabletop Role-Playing Games (TRPGs). GMs play a crucial role in overseeing information, estimating players' intentions, providing environment descriptions, and offering feedback, compensating for current world model deficiencies. To facilitate future explorations for complex interactions, we introduce a benchmark named Tachikuma, comprising a Multiple character and novel Object based interaction Estimation (MOE) task and a supporting dataset. MOE challenges models to understand characters' intentions and accurately determine their actions within intricate contexts involving multi-character and novel object interactions. Besides, the dataset captures log data from real-time communications during gameplay, providing diverse, grounded, and complex interactions for further explorations. Finally, we present a simple prompting baseline and evaluate its performance, demonstrating its effectiveness in enhancing interaction understanding. We hope that our dataset and task will inspire further research in complex interactions with natural language, fostering the development of more advanced AI agents.

MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

Language Urban Odyssey: A Serious Game for Enhancing Second Language Acquisition Through Large Language Models

AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game

LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

Who is Undercover? Guiding LLMs to Explore Multi-Perspective Team Tactic in the Game

Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games

Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash

On the Decision-Making Abilities in Role-Playing using Large Language Models

Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Models

InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context

Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information

RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View

Human Simulacra: Benchmarking the Personification of Large Language Models

Large Language Models Need Consultants for Reasoning: Becoming an Expert in a Complex Human System Through Behavior Simulation

Theory of Mind for Multi-Agent Collaboration via Large Language Models

LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation

User Behavior Simulation with Large Language Model based Agents

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models