Abstract:Quality Diversity (QD) has emerged as a powerful alternative optimization paradigm that aims at generating large and diverse collections of solutions, notably with its flagship algorithm MAP-ELITES (ME) which evolves solutions through mutations and crossovers. While very effective for some unstructured problems, early ME implementations relied exclusively on random search to evolve the population of solutions, rendering them notoriously sample-inefficient for high-dimensional problems, such as when evolving neural networks. Follow-up works considered exploiting gradient information to guide the search in order to address these shortcomings through techniques borrowed from either Black-Box Optimization (BBO) or Reinforcement Learning (RL). While mixing RL techniques with ME unlocked state-of-the-art performance for robotics control problems that require a good amount of exploration, it also plagued these ME variants with limitations common among RL algorithms that ME was free of, such as hyperparameter sensitivity, high stochasticity as well as training instability, including when the population size increases as some components are shared across the population in recent approaches. Furthermore, existing approaches mixing ME with RL tend to be tied to a specific RL algorithm, which effectively prevents their use on problems where the corresponding RL algorithm fails. To address these shortcomings, we introduce a flexible framework that allows the use of any RL algorithm and alleviates the aforementioned limitations by evolving populations of agents (whose definition include hyperparameters and all learnable parameters) instead of just policies. We demonstrate the benefits brought about by our framework through extensive numerical experiments on a number of robotics control problems, some of which with deceptive rewards, taken from the QD-RL literature.

Monte Carlo Elites: Quality-Diversity Selection as a Multi-Armed Bandit Problem

BOP-Elites, a Bayesian Optimisation algorithm for Quality-Diversity search

Bayesian Optimisation for Quality Diversity Search With Coupled Descriptor Functions

BOP-Elites, a Bayesian Optimisation Approach to Quality Diversity Search with Black-Box descriptor functions

Quality with Just Enough Diversity in Evolutionary Policy Search

Quality Diversity for Multi-task Optimization

Efficient Quality-Diversity Optimization through Diverse Quality Species

Dynamic Quality-Diversity Search

Controllable Exploration of a Design Space via Interactive Quality Diversity

Multi-objective Optimization-based Selection for Quality-Diversity by Non-surrounded-dominated Sorting.

Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space

Assessing Quality-Diversity Neuro-Evolution Algorithms Performance in Hard Exploration Problems

Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains

Quality-Diversity Algorithms Can Provably Be Helpful for Optimization

Exploration and Exploitation in Symbolic Regression using Quality-Diversity and Evolutionary Strategies Algorithms

Synergizing Quality-Diversity with Descriptor-Conditioned Reinforcement Learning

Enhancing MAP-Elites with Multiple Parallel Evolution Strategies

Illuminating the Diversity-Fitness Trade-Off in Black-Box Optimization

Design and Analysis of Diversity-Based Parent Selection Schemes for Speeding Up Evolutionary Multi-objective Optimisation

Evolving Populations of Diverse RL Agents with MAP-Elites

Quality Diversity Evolutionary Learning of Decision Trees