Abstract:Large language models (LLMs) have demonstrated impressive capabilities in role-playing tasks. However, there is limited research on whether LLMs can accurately simulate user behavior in real-world scenarios, such as social media. This requires models to effectively analyze a user's history and simulate their role. In this paper, we introduce \textbf{FineRob}, a novel fine-grained behavior simulation dataset. We collect the complete behavioral history of 1,866 distinct users across three social media platforms. Each behavior is decomposed into three fine-grained elements: object, type, and content, resulting in 78.6k QA records. Based on FineRob, we identify two dominant reasoning patterns in LLMs' behavior simulation processes and propose the \textbf{OM-CoT} fine-tuning method to enhance the capability. Through comprehensive experiments, we conduct an in-depth analysis of key factors of behavior simulation and also demonstrate the effectiveness of OM-CoT approach\footnote{Code and dataset are available at \url{<a class="link-external link-https" href="https://github.com/linkseed18612254945/FineRob" rel="external noopener nofollow">this https URL</a>}}

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to use large language models (LLMs) to accurately simulate the fine - grained behaviors of real users on social platforms. Specifically, the researchers are concerned with: 1. **Limitations of existing research**: - Most current research focuses on the performance of large language models in role - playing tasks, but few studies have explored whether these models can accurately simulate user behaviors in complex real - life scenarios (such as social media). - Accurately simulating human behavior, especially in complex real - world scenarios, is a huge challenge. 2. **Proposing new datasets and methods**: - The researchers introduced a new dataset named FineRob, which contains the complete behavior histories of 1,866 different users from three major social platforms (Twitter, Reddit, and Zhihu), with a total of 786,000 fine - grained behavior Q&A records. - Each behavior is decomposed into three fine - grained elements: object, type, and content, in order to more precisely simulate user behavior. 3. **Analyzing the behavior - simulation reasoning patterns of large language models**: - By using the FineRob dataset, the researchers evaluated nine widely - used large language models and discovered two main reasoning patterns: "reasoning based on role stereotypes" and "reasoning based on observation and memory". - Based on this finding, they proposed a new fine - tuning method OM - CoT, which aims to enhance the model's behavior - simulation ability by explicitly integrating observation and memory analysis into the reasoning process. 4. **Experimental verification**: - The researchers verified the effectiveness of the OM - CoT method through a series of experiments, and the results showed that this method significantly improved performance in all three behavior - element simulation tasks. In summary, the main objective of this paper is to explore and improve the ability of large language models in simulating the behaviors of real - life social - platform users, especially the accuracy in fine - grained behavior simulation. By introducing new datasets and an improved fine - tuning method, the researchers hope to promote further development in this field.

Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media

RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

Thinking Before Speaking: A Role-playing Model with Mindset

Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Roles

The Oscars of AI Theater: A Survey on Role-Playing with Language Models

User Behavior Simulation with Large Language Model based Agents

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

On the Decision-Making Abilities in Role-Playing using Large Language Models

Build An Influential Bot In Social Media Simulations With Large Language Models

Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?

Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

Language Urban Odyssey: A Serious Game for Enhancing Second Language Acquisition Through Large Language Models

Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

Are you in a Masquerade? Exploring the Behavior and Impact of Large Language Model Driven Social Bots in Online Social Networks

Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data

Benchmarking Bias in Large Language Models during Role-Playing

RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Models

Orca: Enhancing Role-Playing Abilities of Large Language Models by Integrating Personality Traits

Simulating Field Experiments with Large Language Models

Human Simulacra: Benchmarking the Personification of Large Language Models

Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models