Abstract:Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field.

Survey of Self-Play in Reinforcement Learning

A Survey on Self-play Methods in Reinforcement Learning

Self-play Reinforcement Learning with Comprehensive Critic in Computer Games

A Comparison of Self-Play Algorithms Under a Generalized Framework

Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

Algorithms in Multi-Agent Systems: A Holistic Perspective from Reinforcement Learning and Game Theory

A Survey of Deep Reinforcement Learning in Video Games

A Survey and Critique of Multiagent Deep Reinforcement Learning

Multi-Agent Reinforcement Learning: A Report on Challenges and Approaches

A Survey on Population-Based Deep Reinforcement Learning

A Survey of Meta-Reinforcement Learning

Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain

Deep Reinforcement Learning: A Brief Survey

A survey on intrinsic motivation in reinforcement learning

Multi-agent Reinforcement Learning: A Comprehensive Survey

Evolutionary Reinforcement Learning: A Survey

Provable Self-Play Algorithms for Competitive Reinforcement Learning

Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey

Automatic Curriculum Learning For Deep RL: A Short Survey

Reinforcement Learning Algorithms: A brief survey