Abstract:We present APT, an advanced Large Language Model (LLM)-driven framework that enables autonomous agents to construct complex and creative structures within the Minecraft environment. Unlike previous approaches that primarily concentrate on skill-based open-world tasks or rely on image-based diffusion models for generating voxel-based structures, our method leverages the intrinsic spatial reasoning capabilities of LLMs. By employing chain-of-thought decomposition along with multimodal inputs, the framework generates detailed architectural layouts and blueprints that the agent can execute under zero-shot or few-shot learning scenarios. Our agent incorporates both memory and reflection modules to facilitate lifelong learning, adaptive refinement, and error correction throughout the building process. To rigorously evaluate the agent's performance in this emerging research area, we introduce a comprehensive benchmark consisting of diverse construction tasks designed to test creativity, spatial reasoning, adherence to in-game rules, and the effective integration of multimodal instructions. Experimental results using various GPT-based LLM backends and agent configurations demonstrate the agent's capacity to accurately interpret extensive instructions involving numerous items, their positions, and orientations. The agent successfully produces complex structures complete with internal functionalities such as Redstone-powered systems. A/B testing indicates that the inclusion of a memory module leads to a significant increase in performance, emphasizing its role in enabling continuous learning and the reuse of accumulated experience. Additionally, the agent's unexpected emergence of scaffolding behavior highlights the potential of future LLM-driven agents to utilize subroutine planning and leverage the emergence ability of LLMs to autonomously develop human-like problem-solving techniques.

Minecraft as an Experimental World for AI in Robotics

Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence Agents

Plan4MC: Skill Reinforcement Learning and Planning for Open-World Minecraft Tasks

DIP-RL: Demonstration-Inferred Preference Learning in Minecraft

The MineRL BASALT Competition on Learning from Human Feedback

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models

Ghost in the Minecraft: Generally Capable Agents for Open-World Environments Via Large Language Models with Text-based Knowledge and Memory

APT: Architectural Planning and Text-to-Blueprint Construction Using Large Language Models for Open-World Agents

Scaling Imitation Learning in Minecraft

MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft

OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks

Probabilistic Modeling of Human Teams to Infer False Beliefs

Mastering Diverse Domains through World Models

ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting

Odyssey: Empowering Minecraft Agents with Open-World Skills

Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction

ParliRobo: Participant Lightweight AI Robots for Massively Multiplayer Online Games (Mmogs)

Using Features at Multiple Temporal and Spatial Resolutions to Predict Human Behavior in Real Time