Abstract:Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g.\ DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ${\raise.17ex\hbox{$\scriptstyle\sim$}}$4 hours on one desktop or ${\raise.17ex\hbox{$\scriptstyle\sim$}}$1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.

Deep Learning-Based Operators for Evolutionary Algorithms

Deep Neural Crossover

Deep Reinforcement Learning Based Adaptive Operator Selection for Evolutionary Multi-Objective Optimization

Direct Mutation and Crossover in Genetic Algorithms Applied to Reinforcement Learning Tasks

Neural network crossover in genetic algorithms using genetic programming

Dynamic Selection of Evolutionary Operators Based on Online Learning and Fitness Landscape Analysis

Genetic Algorithms for Evolving Deep Neural Networks

A Melting Pot of Evolution and Learning

Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

Learning to Mutate for Differential Evolution

On Evolutionary Strategy Based on Hybrid Crossover Operators

Learning to select the recombination operator for derivative-free optimization

Learning Matrices of Evolutionary Operators in Genetic Algorithm

A New Crossover Operator and Its Application to Artificial Neural Networks Evolution

Self-adaptation of Genetic Operators Through Genetic Programming Techniques

Evolutionary Generative Adversarial Networks Based on New Fitness Function and Generic Crossover Operator.

An Adaptive Co-evolutionary Algorithm Based on Genotypic Diversity Measure

An Adaptive Neighborhood-Based Multi-Parent Crossover Operator For Real-Coded Genetic Algorithms

An Empirical Study of Genetic Operators in Genetic Algorithms.

Several Strategies of Evolutionary Direction Operator for Genetic Algorithm

Co-Evolutionary Algorithms Based on Mixed Strategy