Abstract:Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g.\ DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ${\raise.17ex\hbox{$\scriptstyle\sim$}}$4 hours on one desktop or ${\raise.17ex\hbox{$\scriptstyle\sim$}}$1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.

Generative Adversarial Neuroevolution for Control Behaviour Imitation

Neuroevolution of Recurrent Architectures on Control Tasks

Generalize Robot Learning from Demonstration to Variant Scenarios with Evolutionary Policy Gradient

An electronic-game framework for evaluating coevolutionary algorithms

A Neuroevolution Approach to General Atari Game Playing

Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

Imitating by Generating: Deep Generative Models for Imitation of Interactive Tasks

An Efficient Application of Neuroevolution for Competitive Multiagent Learning

odNEAT: An Algorithm for Decentralised Online Evolution of Robotic Controllers

Robust Imitation Learning for Automated Game Testing

Towards the Neuroevolution of Low-level artificial general intelligence

Evolution with Opponent-Learning Awareness

Discovering neural policies to drive behaviour by integrating deep reinforcement learning agents with biological neural networks

Generative NeuroEvolution for Deep Learning

Evolutionary Reinforcement Learning via Cooperative Coevolution

Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning

Emergent adaptive behaviour of GRN-controlled simulated robots in a changing environment

A Pragmatic Look at Deep Imitation Learning

Co-Imitation: Learning Design and Behaviour by Imitation

Evolutionarily-Curated Curriculum Learning for Deep Reinforcement Learning Agents

Evolving neural networks to follow trajectories of arbitrary complexity