Abstract:Quality-Diversity algorithms, among which MAP-Elites, have emerged as powerful alternatives to performance-only optimisation approaches as they enable generating collections of diverse and high-performing solutions to an optimisation problem. However, they are often limited to low-dimensional search spaces and deterministic environments. The recently introduced Policy Gradient Assisted MAP-Elites (PGA-MAP-Elites) algorithm overcomes this limitation by pairing the traditional Genetic operator of MAP-Elites with a gradient-based operator inspired by Deep Reinforcement Learning. This new operator guides mutations toward high-performing solutions using policy-gradients. In this work, we propose an in-depth study of PGA-MAP-Elites. We demonstrate the benefits of policy-gradients on the performance of the algorithm and the reproducibility of the generated solutions when considering uncertain domains. We first prove that PGA-MAP-Elites is highly performant in both deterministic and uncertain high-dimensional environments, decorrelating the two challenges it tackles. Secondly, we show that in addition to outperforming all the considered baselines, the collections of solutions generated by PGA-MAP-Elites are highly reproducible in uncertain environments, approaching the reproducibility of solutions found by Quality-Diversity approaches built specifically for uncertain applications. Finally, we propose an ablation and in-depth analysis of the dynamic of the policy-gradients-based variation. We demonstrate that the policy-gradient variation operator is determinant to guarantee the performance of PGA-MAP-Elites but is only essential during the early stage of the process, where it finds high-performing regions of the search space.

BOP-Elites, a Bayesian Optimisation Approach to Quality Diversity Search with Black-Box descriptor functions

Bayesian Optimisation for Quality Diversity Search With Coupled Descriptor Functions

BOP-Elites, a Bayesian Optimisation algorithm for Quality-Diversity search

Quality Diversity for Multi-task Optimization

Efficient Quality-Diversity Optimization through Diverse Quality Species

Quality-Diversity Algorithms Can Provably Be Helpful for Optimization

Dynamic Quality-Diversity Search

Monte Carlo Elites: Quality-Diversity Selection as a Multi-Armed Bandit Problem

Multi-objective Optimization-based Selection for Quality-Diversity by Non-surrounded-dominated Sorting.

Finding Near-Optimal Portfolios With Quality-Diversity

Runtime Analysis of Quality Diversity Algorithms

Process Quality Control Through Bayesian Optimization with Adaptive Local Convergence

Quality with Just Enough Diversity in Evolutionary Policy Search

Illuminating the Diversity-Fitness Trade-Off in Black-Box Optimization

Synergizing Quality-Diversity with Descriptor-Conditioned Reinforcement Learning

Quality-Diversity Meta-Evolution: customising behaviour spaces to a meta-objective

Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains

Differentiable Quality Diversity

Quality and Diversity Optimization: A Unifying Modular Framework

Multi-Emitter MAP-Elites: Improving quality, diversity and convergence speed with heterogeneous sets of emitters

Gradient-Informed Quality Diversity for the Illumination of Discrete Spaces