Abstract:Prevailing large language models (LLMs) are capable of human responses simulation through its unprecedented content generation and reasoning abilities. However, it is not clear whether and how to leverage LLMs to simulate field experiments. In this paper, we propose and evaluate two prompting strategies: the observer mode that allows a direct prediction on main conclusions and the participant mode that simulates distributions of responses from participants. Using this approach, we examine fifteen well cited field experimental papers published in INFORMS and MISQ, finding encouraging alignments between simulated experimental results and the actual results in certain scenarios. We further identify topics of which LLMs underperform, including gender difference and social norms related research. Additionally, the automatic and standardized workflow proposed in this paper enables the possibility of a large-scale screening of more papers with field experiments. This paper pioneers the utilization of large language models (LLMs) for simulating field experiments, presenting a significant extension to previous work which focused solely on lab environments. By introducing two novel prompting strategies, observer and participant modes, we demonstrate the ability of LLMs to both predict outcomes and replicate participant responses within complex field settings. Our findings indicate a promising alignment with actual experimental results in certain scenarios, achieving a stimulation accuracy of 66% in observer mode. This study expands the scope of potential applications for LLMs and illustrates their utility in assisting researchers prior to engaging in expensive field experiments. Moreover, it sheds light on the boundaries of LLMs when used in simulating field experiments, serving as a cautionary note for researchers considering the integration of LLMs into their experimental toolkit.

Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media

RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

Thinking Before Speaking: A Role-playing Model with Mindset

Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Roles

The Oscars of AI Theater: A Survey on Role-Playing with Language Models

User Behavior Simulation with Large Language Model based Agents

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

When Large Language Model based Agent Meets User Behavior Analysis: A Novel User Simulation Paradigm

User Behavior Simulation with Large Language Model-based Agents for Recommender Systems

On the Decision-Making Abilities in Role-Playing using Large Language Models

Build An Influential Bot In Social Media Simulations With Large Language Models

Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?

Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

Language Urban Odyssey: A Serious Game for Enhancing Second Language Acquisition Through Large Language Models

Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

Are you in a Masquerade? Exploring the Behavior and Impact of Large Language Model Driven Social Bots in Online Social Networks

Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data

Benchmarking Bias in Large Language Models during Role-Playing

RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Models

Orca: Enhancing Role-Playing Abilities of Large Language Models by Integrating Personality Traits

Simulating Field Experiments with Large Language Models