Abstract:Agent-based models (ABMs) stand as an essential paradigm for proposing and validating hypothetical solutions or policies aimed at addressing challenges posed by complex systems and achieving various objectives. This process demands labor-intensive endeavors and multidisciplinary expertise. Large language models (LLMs) encapsulating cross-domain knowledge and programming proficiency could potentially alleviate the difficulty of this process. However, LLMs excel in handling sequential information, making it challenging for analyzing the intricate interactions and nonlinear dynamics inherent in ABMs. Additionally, due to the lack of self-evaluation capability of LLMs, relying solely on LLMs is insufficient to effectively accomplish this process. In this paper, we present SAGE, a general solution-oriented ABM generation framework designed for automatic modeling and generating solutions for targeted problems. Unlike approaches reliant on expert handcrafting or resource-intensive neural network training, SAGE establishes a verifier-assisted iterative in-context learning process employing large language models (LLMs) to leverages their inherent cross-domain knowledge for tackling intricate demands from diverse domain scenarios. In SAGE, we introduce an semi-structured conceptual representation expliciting the intricate structures of ABMs and an objective representation to guide LLMs in modeling scenarios and proposing hypothetical solutions through in-context learning. To ensure the model executability and solution feasibility, SAGE devises a two-level verifier with chain-of-thought prompting tailored to the complex interactions and non-linear dynamics of ABMs, driving the iterative generation optimization. Moreover, we construct an evaluation dataset of solution-oriented ABMs from open sources.It contains practical models across various domains.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use large - language models (LLMs) to automatically generate executable and effective agent - based models (ABMs) to solve specific real - world problems. Specifically, the paper points out that although LLMs perform well in processing sequential information, they face challenges in generating, analyzing, and optimizing ABMs because ABMs involve complex interactions and non - linear dynamics. In addition, LLMs lack self - assessment capabilities, resulting in the generated ABMs may contain factual errors, logical inconsistencies, or over - simplification problems. To overcome these challenges, the paper proposes a framework named SAGE (Solution - oriented Agent - based models GEneration). SAGE aims to generate high - quality solution - oriented ABMs by combining the context - learning ability of LLMs and two - layer validators. The workflow of SAGE is divided into two stages: 1. **Modeling stage**: The goal of this stage is to generate an executable ABM according to the given problem scenario. The user provides a conceptual representation of the scenario, and the LLM generates an ABM program. The generated ABM is then automatically verified by the first - level verifier (Verifier - level1) to ensure its executability and completeness. If the verification fails, the LLM iteratively modifies the program according to the verification results until a correct ABM is generated. 2. **Solving stage**: The focus of this stage is to generate effective solutions for specific problems. The user uses the target representation to specify the target problem and its required effects or constraints. The second - level verifier (Verifier - level2) uses the LLM to generate verification functions, which are used to check whether the simulation results meet specific targets, thereby driving the optimization of the solution. The verification results guide the LLM to gradually generate solutions, including relationship extraction, cause analysis, and solution proposals. The generated solutions are integrated back into the existing ABM and further iteratively modified by the first - level verifier until the ABM contains the new solution and is executable and comprehensive. In this way, SAGE can not only use the powerful capabilities of LLMs to generate ABMs, but also ensure the quality and effectiveness of the generated models and solutions, so as to better cope with the challenges of complex systems.

Solution-oriented Agent-based Models Generation with Verifier-assisted Iterative In-context Learning

Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulations

SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text

Adversarial Multi-Agent Evaluation of Large Language Models through Iterative Debates

An Evaluation-Driven Approach to Designing LLM Agents: Process and Architecture

AgentBench: Evaluating LLMs as Agents

Navigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMs

Large Language Model Evaluation Via Multi AI Agents: Preliminary results

AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

AgentSims: An Open-Source Sandbox for Large Language Model Evaluation

LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery

The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests

MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification

Large Language Models as Planning Domain Generators

Autonomous Multi-Objective Optimization Using Large Language Model

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

Iterative Data Generation with Large Language Models for Aspect-based Sentiment Analysis

Model Management Agent for Expensive Constrained Multi-Objective Optimization

RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidance

Optimal Decision Making Through Scenario Simulations Using Large Language Models