Abstract:We propose a novel approach to the statistical analysis of stochastic simulation models and, especially, agent-based models (ABMs). Our main goal is to provide fully automated, model-independent and tool-supported techniques and algorithms to inspect simulations and perform counterfactual analysis. Our approach: (i) is easy-to-use by the modeller, (ii) improves reproducibility of results, (iii) optimizes running time given the modeller's machine, (iv) automatically chooses the number of required simulations and simulation steps to reach user-specified statistical confidence, and (v) automates a variety of statistical tests. In particular, our techniques are designed to distinguish the transient dynamics of the model from its steady-state behaviour (if any), estimate properties in both 'phases', and provide indications on the (non-)ergodic nature of the simulated processes - which, in turn, allows one to gauge the reliability of a steady-state analysis. Estimates are equipped with statistical guarantees, allowing for robust comparisons across computational experiments. To demonstrate the effectiveness of our approach, we apply it to two models from the literature: a large-scale macro-financial ABM and a small scale prediction market model. Compared to prior analyses of these models, we obtain new insights and we are able to identify and fix some erroneous conclusions.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the degree of automation and reliability of statistical analysis of Agent - Based Models (ABMs). Specifically, the author proposes a new method, aiming to provide a fully automated, model - independent, tool - supported technique and algorithm to check simulation results and perform counterfactual analysis. The goals of this method include: 1. **Ease of use**: Enable modelers to use these techniques conveniently. 2. **Reproducibility of results**: Improve the reproducibility of research results. 3. **Optimization of running time**: Optimize the simulation running time according to the modeler's machine. 4. **Automatic selection of simulation times and steps**: Automatically determine the number of simulation times and steps required to reach the user - specified statistical confidence level. 5. **Automation of multiple statistical tests**: Automatically perform multiple statistical tests, which are often overlooked in practice. In addition, this method is specifically designed to distinguish the transient dynamics and steady - state behavior (if any) of the model, estimate the properties of these two phases, and provide an indication of the (non - ) ergodicity of the simulation process. This helps to assess the reliability of the steady - state analysis. The estimates come with statistical guarantees, allowing for robust comparison in different computational experiments. To verify the effectiveness of this method, the author applies it to two models in the literature: a large - scale macro - financial ABM and a small - scale prediction market model. Compared with previous analyses, the author obtains new insights and is able to identify and correct some wrong conclusions.

Automated and Distributed Statistical Analysis of Economic Agent-Based Models

Agent‐based models in sociology

Economic evaluations with agent-based modelling: an introduction

On learning agent-based models from data

A bright future for financial agent-based models

Sensitivity analysis of agent-based models: a new protocol

EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities

Using machine learning as a surrogate model for agent-based simulations

Adaptive Agents and Data Quality in Agent-Based Financial Markets

A Comparison of Economic Agent-Based Model Calibration Methods

Scalable Agent-Based Modeling for Complex Financial Market Simulations

Social Network Analysis and Validation of an Agent-Based Model

An Agent-Based Model With Realistic Financial Time Series: A Method for Agent-Based Models Validation

Noise-free comparison of stochastic agent-based simulations using common random numbers

ABIDES-Economist: Agent-Based Simulation of Economic Systems with Learning Agents

Quantitatively Assessing the Benefits of Model-driven Development in Agent-based Modeling and Simulation

A Bibliometric Analysis on Agent-Based Models in Finance: Identification of Community Clusters and Future Research Trends

Tutorial on agent-based modeling and simulation

Simulating the Economic Impact of Rationality through Reinforcement Learning and Agent-Based Modelling

New approaches in agent-based modeling of complex financial systems

Dealing with uncertainty in agent-based models for short-term predictions