Abstract:Many recent works have explored using language models for planning problems. One line of research focuses on translating natural language descriptions of planning tasks into structured planning languages, such as the planning domain definition language (PDDL). While this approach is promising, accurately measuring the quality of generated PDDL code continues to pose significant challenges. First, generated PDDL code is typically evaluated using planning validators that check whether the problem can be solved with a planner. This method is insufficient because a language model might generate valid PDDL code that does not align with the natural language description of the task. Second, existing evaluation sets often have natural language descriptions of the planning task that closely resemble the ground truth PDDL, reducing the challenge of the task. To bridge this gap, we introduce \benchmarkName, a benchmark designed to evaluate language models' ability to generate PDDL code from natural language descriptions of planning tasks. We begin by creating a PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL code generated by language models by flexibly comparing it against a ground truth PDDL. Then, we present a dataset of $132,037$ text-to-PDDL pairs across 13 different tasks, with varying levels of difficulty. Finally, we evaluate several API-access and open-weight language models that reveal this task's complexity. For example, $87.6\%$ of the PDDL problem descriptions generated by GPT-4o are syntactically parseable, $82.2\%$ are valid, solve-able problems, but only $35.1\%$ are semantically correct, highlighting the need for a more rigorous benchmark for this problem.

Playing Angry Birds with a Domain-Independent PDDL+ Planner

Deep Q-Network for Angry Birds

The Computational Complexity of Angry Birds

Real-World Planning with PDDL+ and Beyond

Procedural Generation of Angry Birds Levels using Building Constructive Grammar with Chinese-Style and/or Japanese-Style Models

On Solving the Rubik's Cube with Domain-Independent Planners Using Standard Representations

Utilizing Generative Adversarial Networks for Stable Structure Generation in Angry Birds

PROC2PDDL: Open-Domain Planning Representations from Texts

Planning by Rewriting

PDDLEGO: Iterative Planning in Textual Environments

Long-Term Planning with Deep Reinforcement Learning on Autonomous Drones

Hierarchical Object-Oriented POMDP Planning for Object Rearrangement

Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning

Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models

Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents.

Learning adaptive planning representations with natural language guidance

A Good Snowman is Hard to Plan

Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages

SwarmPlay: Interactive Tic-tac-toe Board Game with Swarm of Nano-UAVs driven by Reinforcement Learning

Points2Plans: From Point Clouds to Long-Horizon Plans with Composable Relational Dynamics

Plotting: a case study in lifted planning with constraints