Abstract:The ability to autonomously explore and resolve tasks with minimal human guidance is crucial for the self-development of embodied intelligence. Although reinforcement learning methods can largely ease human effort, it's challenging to design reward functions for real-world tasks, especially for high-dimensional robotic control, due to complex relationships among joints and tasks. Recent advancements large language models (LLMs) enable automatic reward function design. However, approaches evaluate reward functions by re-training policies from scratch placing an undue burden on the reward function, expecting it to be effective throughout the whole policy improvement process. We argue for a more practical strategy in robotic autonomy, focusing on refining existing policies with policy-dependent reward functions rather than a universal one. To this end, we propose a novel reward-policy co-evolution framework where the reward function and the learned policy benefit from each other's progressive on-the-fly improvements, resulting in more efficient and higher-performing skill acquisition. Specifically, the reward evolution process translates the robot's previous best reward function, descriptions of tasks and environment into text inputs. These inputs are used to query LLMs to generate a dynamic amount of reward function candidates, ensuring continuous improvement at each round of evolution. For policy evolution, our method generates new policy populations by hybridizing historically optimal and random policies. Through an improved Bayesian optimization, our approach efficiently and robustly identifies the most capable and plastic reward-policy combination, which then proceeds to the next round of co-evolution. Despite using less data, our approach demonstrates an average normalized improvement of 95.3% across various high-dimensional robotic skill learning tasks.

Evolutionary Reinforcement Learning via Cooperative Coevolution

Evolutionary reinforcement learning via cooperative coevolutionary negatively correlated search

Multi-Agent Evolutionary Reinforcement Learning Based on Cooperative Games

Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution

Proximal Distilled Evolutionary Reinforcement Learning

Evolving Constrained Reinforcement Learning Policy

Evolution-Guided Policy Gradient in Reinforcement Learning

ERL-Re$^2$: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation

Adaptive Evolutionary Reinforcement Learning with Policy Direction

Balance of exploration and exploitation: Non-cooperative game-driven evolutionary reinforcement learning

Deep Reinforcement Learning Assisted Co-Evolutionary Differential Evolution for Constrained Optimization.

Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation

Evolutionary Reinforcement Learning: A Systematic Review and Future Directions

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

Evolutionary Diversity Optimization with Clustering-based Selection for Reinforcement Learning

Evolutionary Algorithms for Reinforcement Learning

Neuroevolution of Recurrent Architectures on Control Tasks

Evolution of Cooperative Ensemble Neural Network Controller for Autonomous Mobile Robots

Evolutionary Reinforcement Learning: A Survey

Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts

EvIL: Evolution Strategies for Generalisable Imitation Learning