Abstract:Large Language Models (LLMs) have the capacity of performing complex scheduling in a multi-agent system and can coordinate these agents into completing sophisticated tasks that require extensive collaboration. However, despite the introduction of numerous gaming frameworks, the community has insufficient benchmarks towards building general multi-agents collaboration infrastructure that encompass both LLM and human-NPCs collaborations. In this work, we propose a novel infrastructure - MindAgent - to evaluate planning and coordination emergent capabilities for gaming interaction. In particular, our infrastructure leverages existing gaming framework, to i) require understanding of the coordinator for a multi-agent system, ii) collaborate with human players via un-finetuned proper instructions, and iii) establish an in-context learning on few-shot prompt with feedback. Furthermore, we introduce CUISINEWORLD, a new gaming scenario and related benchmark that dispatch a multi-agent collaboration efficiency and supervise multiple agents playing the game simultaneously. We conduct comprehensive evaluations with new auto-metric CoS for calculating the collaboration efficiency. Finally, our infrastructure can be deployed into real-world gaming scenarios in a customized VR version of CUISINEWORLD and adapted in existing broader Minecraft gaming domain. We hope our findings on LLMs and the new infrastructure for general-purpose scheduling and coordination can help shed light on how such skills can be obtained by learning from large language corpora.

LevelMerge: Collaborative Game Level Editing by Merging Labeled Graphs

On Linking Level Segments

Integrating Automated Play in Level Co-Creation

Game Level Blending using a Learned Level Representation

Learning to Blend Computer Game Levels

Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators

Lode Enhancer: Level Co-creation Through Scaling

Joint Level Generation and Translation Using Gameplay Videos

A Dialogue Game for Eliciting Balanced Collaboration

A Novel CNet-assisted Evolutionary Level Repairer and Its Applications to Super Mario Bros

Improving Deep Localized Level Analysis: How Game Logs Can Help

EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing

Toward Game Level Generation from Gameplay Videos

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

Level Generation Through Large Language Models

An Approach to Sharing Legacy TV/Arcade Games for Real-Time Collaboration

Learning Constructive Primitives for Online Level Generation and Real-time Content Adaptation in Super Mario Bros

Level Assembly as a Markov Decision Process

Hierarchically Composing Level Generators for the Creation of Complex Structures

A Framework for Leveraging Human Computation Gaming to Enhance Knowledge Graphs for Accuracy Critical Generative AI Applications

MindAgent: Emergent Gaming Interaction