Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

John Chen,Xi Lu,Michael Rejtig,David Du,Ruth Bagley,Michael S. Horn,Uri J. Wilensky

DOI: https://doi.org/10.48550/arXiv.2401.17163

IF: 6.4588

2024-01-30

Human-Computer Interaction

Abstract:Large Language Models (LLMs) have the potential to fundamentally change the way people engage in computer programming. Agent-based modeling (ABM) has become ubiquitous in natural and social sciences and education, yet no prior studies have explored the potential of LLMs to assist it. We designed NetLogo Chat to support the learning and practice of NetLogo, a programming language for ABM. To understand how users perceive, use, and need LLM-based interfaces, we interviewed 30 participants from global academia, industry, and graduate schools. Experts reported more perceived benefits than novices and were more inclined to adopt LLMs in their workflow. We found significant differences between experts and novices in their perceptions, behaviors, and needs for human-AI collaboration. We surfaced a knowledge gap between experts and novices as a possible reason for the benefit gap. We identified guidance, personalization, and integration as major needs for LLM-based interfaces to support the programming of ABM.

What problem does this paper attempt to address?

The paper aims to explore the potential application of Large Language Models (LLMs) in Agent-Based Modeling (ABM) and supports users in learning and practicing the NetLogo programming language by designing a new interface called NetLogo Chat. NetLogo is a programming language widely used for ABM, capable of simulating complex system phenomena through simple rules. The study interviewed 30 expert and novice users from academia, industry, and graduate schools worldwide, comparing their different perceptions, behaviors, and needs regarding the LLM-driven interface in open modeling tasks.

The main objectives include:
1. Exploring how expert and novice users perceive the support of the LLM-driven interface for their NetLogo learning and practice, including its advantages, disadvantages, and adoption plans.
2. Understanding how expert and novice users utilize the LLM-driven interface to support their NetLogo learning and practice.
3. Identifying the specific needs of expert and novice users for the LLM interface to support their NetLogo learning and practice.

The study found that expert users are more aware of the potential benefits brought by LLMs compared to novice users and are more inclined to incorporate them into their workflows. There are significant differences in perceptions, behaviors, and needs between experts and novices, which may stem from the knowledge gap between the two groups. Additionally, the study highlights three key themes—guidance, personalization, and integration—as focal points for future LLM interface design to better support ABM programming.