Abstract:Programmatic reinforcement learning (PRL) has been explored for representing policies through programs as a means to achieve interpretability and generalization. Despite promising outcomes, current state-of-the-art PRL methods are hindered by sample inefficiency, necessitating tens of millions of program-environment interactions. To tackle this challenge, we introduce a novel LLM-guided search framework (LLM-GS). Our key insight is to leverage the programming expertise and common sense reasoning of LLMs to enhance the efficiency of assumption-free, random-guessing search methods. We address the challenge of LLMs' inability to generate precise and grammatically correct programs in domain-specific languages (DSLs) by proposing a Pythonic-DSL strategy - an LLM is instructed to initially generate Python codes and then convert them into DSL programs. To further optimize the LLM-generated programs, we develop a search algorithm named Scheduled Hill Climbing, designed to efficiently explore the programmatic search space to improve the programs consistently. Experimental results in the Karel domain demonstrate our LLM-GS framework's superior effectiveness and efficiency. Extensive ablation studies further verify the critical role of our Pythonic-DSL strategy and Scheduled Hill Climbing algorithm. Moreover, we conduct experiments with two novel tasks, showing that LLM-GS enables users without programming skills and knowledge of the domain or DSL to describe the tasks in natural language to obtain performant programs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to improve the sample efficiency of Programmatic Reinforcement Learning (PRL) to make it closer to the requirements of practical applications?** Specifically, although current PRL methods can achieve higher interpretability and generalization ability through programmatic strategies, their main problem is **low sample efficiency**, requiring tens of millions of program - environment interactions to find effective solutions. This inefficiency limits the application of PRL in real - world decision - making problems. To solve this problem, the paper proposes a search framework guided by large language models (LLM - GS) and improves PRL in the following ways: 1. **Utilize the knowledge and reasoning ability of LLM**: By designing task - aware prompts, let LLM generate initial programs, thereby reducing the time for random guessing. 2. **Pythonic - DSL strategy**: Since there are syntactic and semantic difficulties for LLM to directly generate domain - specific language (DSL) programs, the paper proposes to let LLM generate Python code first and then convert it into DSL programs. 3. **Scheduled Hill Climbing algorithm**: To further optimize the programs generated by LLM, the paper designs an efficient search algorithm that gradually improves program performance by dynamically adjusting the size of the search neighborhood. Ultimately, the goal of the paper is to significantly reduce the number of environment interactions while maintaining high rewards through the LLM - GS framework, thereby improving the sample efficiency of PRL. --- ### Formula Summary One of the key formulas used in the paper is the function for scheduling the size of the search neighborhood: $$ \log_2 k(n)=(1 - r(n)) \log_2 K_{\text{start}}+r(n) \log_2 K_{\text{end}} $$ where: - $n$ represents the number of evaluated programs; - $k(n)$ is the number of current neighborhood programs; - $K_{\text{start}}$ and $K_{\text{end}}$ represent the initial and final neighborhood search numbers respectively; - $r(n)$ is a sinusoidal function that gradually increases from 0 to 1, controlling the linear proportion. Through this formula, the size of the search neighborhood will gradually increase as the execution budget is used, thereby achieving more efficient exploration and optimization.

Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search

Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards

Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs

Secrets of RLHF in Large Language Models Part I: PPO

Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

Large Language Models as Generalizable Policies for Embodied Tasks

Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks

Synthesizing Interpretable Control Policies through Large Language Model Guided Search

LGR2: Language Guided Reward Relabeling for Accelerating Hierarchical Reinforcement Learning

Code as Policies: Language Model Programs for Embodied Control

LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models

Synthesizing Programmatic Policy for Generalization Within Task Domain

RL-GPT: Integrating Reinforcement Learning and Code-as-policy

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

Reinforcement Learning Problem Solving with Large Language Models

Efficient Reinforcement Learning with Large Language Model Priors

Learning to Generate Better Than Your LLM

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods