Abstract:Reinforcement Learning (RL) offers a fundamental framework for discovering optimal action strategies through interactions within unknown environments. Recent advancement have shown that the performance and applicability of RL can significantly be enhanced by exploiting a population of agents in various ways. Zeroth-Order Optimization (ZOO) leverages an agent population to estimate the gradient of the objective function, enabling robust policy refinement even in non-differentiable scenarios. As another application, Genetic Algorithms (GA) boosts the exploration of policy landscapes by mutational generation of policy diversity in an agent population and its refinement by selection. A natural question is whether we can have the best of two worlds that the agent population can have. In this work, we propose Ancestral Reinforcement Learning (ARL), which synergistically combines the robust gradient estimation of ZOO with the exploratory power of GA. The key idea in ARL is that each agent within a population infers gradient by exploiting the history of its ancestors, i.e., the ancestor population in the past, while maintaining the diversity of policies in the current population as in GA. We also theoretically reveal that the populational search in ARL implicitly induces the KL-regularization of the objective function, resulting in the enhanced exploration. Our results extend the applicability of populational algorithms for RL.

RLGA: A Reinforcement Learning Based Genetic Algorithm

Evolutionary Algorithm of Port Based Location Routing Problem

SCGA: Controlling Genetic Algorithms with Sarsa(0)

A Study of Compulsive Genetic Algorithm and Its Performance

Deep Reinforcement Learning using Genetic Algorithm for Parameter Optimization

Direct Mutation and Crossover in Genetic Algorithms Applied to Reinforcement Learning Tasks

Genetic algorithm for reinforcement learning

An Improved Real-Coded Genetic Algorithm and Its Application

Multiple Policy Selection Genetic Algorithm Based on Reinforcement Learning

GARA: A novel approach to Improve Genetic Algorithms' Accuracy and Efficiency by Utilizing Relationships among Genes

Recursive Learning of Genetic Algorithms with Task Decomposition and Varied Rule Set

Nonlinear mapping using real-valued genetic algorithm

Improved Genetic Algorithm for Robotic Cell Scheduling Problem with Flexible Processing Times

Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning

UAV Reconnaissance Task Allocation with Reinforcement Learning and Genetic Algorithm

Coordinating Exploration and Exploitation to Construct Genetic Algorithms

A Genetic Algorithm by Coordinating Exploration and Exploitation --Convergence Properties and Performance Analyses

A Novel Genetic Algorithm with Multiple Sub-Population Parallel Search Mechanism.

A Neural Networks for Real Coded Genetic Algorithm

Robustness and Performance of Deep Reinforcement Learning.

A Hybrid Strategy: Real-Coded Genetic Algorithm and Chaotic Search