Abstract:We present APT, an advanced Large Language Model (LLM)-driven framework that enables autonomous agents to construct complex and creative structures within the Minecraft environment. Unlike previous approaches that primarily concentrate on skill-based open-world tasks or rely on image-based diffusion models for generating voxel-based structures, our method leverages the intrinsic spatial reasoning capabilities of LLMs. By employing chain-of-thought decomposition along with multimodal inputs, the framework generates detailed architectural layouts and blueprints that the agent can execute under zero-shot or few-shot learning scenarios. Our agent incorporates both memory and reflection modules to facilitate lifelong learning, adaptive refinement, and error correction throughout the building process. To rigorously evaluate the agent's performance in this emerging research area, we introduce a comprehensive benchmark consisting of diverse construction tasks designed to test creativity, spatial reasoning, adherence to in-game rules, and the effective integration of multimodal instructions. Experimental results using various GPT-based LLM backends and agent configurations demonstrate the agent's capacity to accurately interpret extensive instructions involving numerous items, their positions, and orientations. The agent successfully produces complex structures complete with internal functionalities such as Redstone-powered systems. A/B testing indicates that the inclusion of a memory module leads to a significant increase in performance, emphasizing its role in enabling continuous learning and the reuse of accumulated experience. Additionally, the agent's unexpected emergence of scaffolding behavior highlights the potential of future LLM-driven agents to utilize subroutine planning and leverage the emergence ability of LLMs to autonomously develop human-like problem-solving techniques.

Nebula: A discourse aware Minecraft Builder

Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft

Improving Agent Interactions in Virtual Environments with Language Models

Large Language Models as Minecraft Agents

Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning

Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling

MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Collaborative Learning

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World

OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents

APT: Architectural Planning and Text-to-Blueprint Construction Using Large Language Models for Open-World Agents

Reasoning About Pragmatics with Neural Listeners and Speakers

Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback

A LLM Benchmark based on the Minecraft Builder Dialog Agent Task

Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Tasks

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Nebula: A Coordinating Grammar of Graphics

Human Latency Conversational Turns for Spoken Avatar Systems

CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society

ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting

Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement