Abstract:Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field.

A novelty-search-based evolutionary reinforcement learning algorithm for continuous optimization problems

Efficient Novelty Search Through Deep Reinforcement Learning

Dynamics-aware Novelty Search with Behavior Repulsion

Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents

An Adaptive Real Coded Population-Based Incremental Learning Algorithm for Design Optimizations in Continuous Space

Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities

Evolutionary Reinforcement Learning: A Survey

Maximum Entropy Reinforcement Learning with Evolution Strategies

Evolutionary Reinforcement Learning via Cooperative Coevolution

Auto-configuring Exploration-Exploitation Tradeoff in Evolutionary Computation via Deep Reinforcement Learning

Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms

Balance of exploration and exploitation: Non-cooperative game-driven evolutionary reinforcement learning

Evolutionary Reinforcement Learning: A Systematic Review and Future Directions

Deep reinforcement learning assisted novelty search in Voronoi regions for constrained multi-objective optimization

Evolution of Swarm Robotics Systems with Novelty Search

Evolutionary reinforcement learning via cooperative coevolutionary negatively correlated search

Efficacy of Modern Neuro-Evolutionary Strategies for Continuous Control Optimization

Evolution-Guided Policy Gradient in Reinforcement Learning

Evolutionary Diversity Optimization with Clustering-based Selection for Reinforcement Learning

BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization

Evolving Constrained Reinforcement Learning Policy