Discrete Natural Evolution Strategies

Ahmad Ayaz Amin

2024-03-30

Abstract:Natural evolution strategies are a class of approximate-gradient black-box optimizers that have been successfully used for continuous parameter spaces. In this paper, we derive NES algorithms for discrete parameter spaces and demonstrate their effectiveness in tasks involving discrete parameters.

Neural and Evolutionary Computing,Optimization and Control

What problem does this paper attempt to address?

Natural Evolution Strategy for Mixed-Integer Black-Box Optimization

Koki Ikeda,Isao Ono

DOI: https://doi.org/10.1145/3583131.3590518

2023-04-21

Abstract:This paper proposes a natural evolution strategy (NES) for mixed-integer black-box optimization (MI-BBO) that appears in real-world problems such as hyperparameter optimization of machine learning and materials design. This problem is difficult to optimize because plateaus where the values do not change appear when the integer variables are relaxed to the continuous ones. CMA-ES w. Margin that addresses the plateaus reportedly showed good performance on MI-BBO benchmark problems. However, it has been observed that the search performance of CMA-ES w. Margin deteriorates when continuous variables contribute more to the objective function value than integer ones. In order to address the problem of CMA-ES w. Margin, we propose Distance-weighted eXponential Natural Evolution Strategy taking account of Implicit Constraint and Integer (DX-NES-ICI). We compare the search performance of DX-NES-ICI with that of CMA-ES w. Margin through numerical experiments. As a result, DX-NES-ICI was up to 3.7 times better than CMA-ES w. Margin in terms of a rate of finding the optimal solutions on benchmark problems where continuous variables contribute more to the objective function value than integer ones. DX-NES-ICI also outperformed CMA-ES w. Margin on problems where CMA-ES w. Margin originally showed good performance.

Neural and Evolutionary Computing
Fast Moving Natural Evolution Strategy for High-Dimensional Problems

Masahiro Nomura,Isao Ono

DOI: https://doi.org/10.48550/arXiv.2201.11422

2022-05-08

Abstract:In this work, we propose a new variant of natural evolution strategies (NES) for high-dimensional black-box optimization problems. The proposed method, CR-FM-NES, extends a recently proposed state-of-the-art NES, Fast Moving Natural Evolution Strategy (FM-NES), in order to be applicable in high-dimensional problems. CR-FM-NES builds on an idea using a restricted representation of a covariance matrix instead of using a full covariance matrix, while inheriting an efficiency of FM-NES. The restricted representation of the covariance matrix enables CR-FM-NES to update parameters of a multivariate normal distribution in linear time and space complexity, which can be applied to high-dimensional problems. Our experimental results reveal that CR-FM-NES does not lose the efficiency of FM-NES, and on the contrary, CR-FM-NES has achieved significant speedup compared to FM-NES on some benchmark problems. Furthermore, our numerical experiments using 200, 600, and 1000-dimensional benchmark problems demonstrate that CR-FM-NES is effective over scalable baseline methods, VD-CMA and Sep-CMA.

Neural and Evolutionary Computing,Machine Learning
Look-ahead natural evolutionary strategies

Jin Liang Jia,Alfredo Alan Flores Saldivar,Lin Li,Yun Li

DOI: https://doi.org/10.1145/3377929.3389998

2020-01-01

Abstract:Natural1 evolution strategy (NES) algorithms have seen a variety of successful applications recently, but still need to improve algorithmic convergence. To address this issue, we propose the use of a look-ahead scheme predictively to estimate the natural acceleration in the algorithm. This estimation is to assist convergence towards the global optimum faster. The resultant algorithm is termed a "Look-Ahead Natural Evolutionary Strategy" (LANES) algorithm. It is tested against a combination of multiple benchmarks using a number of well-known benchmark problems. Compared with the original NES, the LANES increases the optimizer overheads, but offers a steady improvement in optimality, accuracy, and convergence.
Hard-Thresholding Meets Evolution Strategies in Reinforcement Learning

Chengqian Gao,William de Vazelhes,Hualin Zhang,Bin Gu,Zhiqiang Xu

2024-05-03

Abstract:Evolution Strategies (ES) have emerged as a competitive alternative for model-free reinforcement learning, showcasing exemplary performance in tasks like Mujoco and Atari. Notably, they shine in scenarios with imperfect reward functions, making them invaluable for real-world applications where dense reward signals may be elusive. Yet, an inherent assumption in ES, that all input features are task-relevant, poses challenges, especially when confronted with irrelevant features common in real-world problems. This work scrutinizes this limitation, particularly focusing on the Natural Evolution Strategies (NES) variant. We propose NESHT, a novel approach that integrates Hard-Thresholding (HT) with NES to champion sparsity, ensuring only pertinent features are employed. Backed by rigorous analysis and empirical tests, NESHT demonstrates its promise in mitigating the pitfalls of irrelevant features and shines in complex decision-making problems like noisy Mujoco and Atari tasks.

Neural and Evolutionary Computing,Machine Learning
Generative Evolutionary Strategy For Black-Box Optimizations

Changhwi Park

2024-01-27

Abstract:Many scientific and technological problems are related to optimization. Among them, black-box optimization in high-dimensional space is particularly challenging. Recent neural network-based black-box optimization studies have shown noteworthy achievements. However, their capability in high-dimensional search space is still limited. This study proposes a black-box optimization method based on the evolution strategy (ES) and the generative neural network (GNN) model. We designed the algorithm so that the ES and the GNN model work cooperatively. This hybrid model enables reliable training of surrogate networks; it optimizes multi-objective, high-dimensional, and stochastic black-box functions. Our method outperforms baseline optimization methods in this experiment, including ES, and Bayesian optimization.

Neural and Evolutionary Computing
Maximum Entropy Reinforcement Learning with Evolution Strategies

Longxiang Shi,Shijian Li,Qian Zheng,Longbing Cao,Long Yang,Gang Pan

DOI: https://doi.org/10.1109/ijcnn48605.2020.9207570

2020-01-01

Abstract:Evolution strategies (ES) have recently raised attention in solving challenging tasks with low computation costs and high scalability. However, it is well-known that evolution strategies reinforcement learning (RL) methods suffer from low stability. Without careful consideration, ES methods are sensitive to local optima and are unstable in learning. Therefore, there is an urgent need for improving the stability of ES methods in solving RL problems. In this paper, we propose a simple yet efficient ES method to stabilize the learning. Specifically, we propose a framework to incorporate the maximum entropy reinforcement learning with evolution strategies and derive an efficient entropy calculation method for linear policies. We further present a practical algorithm called maximum entropy evolution policy search based on the proposed framework, which is efficient and stable for policy search in continuous control. Our algorithm shows high stability across different random seeds and can obtain comparable results in performance against some existing derivative-free RL methods on several of the well-known benchmark MuJoCo robotic control tasks.
Self-guided Evolution Strategies with Historical Estimated Gradients

Fei-Yu Liu,Zi-Niu Li,Chao Qian

DOI: https://doi.org/10.24963/ijcai.2020/205

2020-01-01

Abstract:Evolution Strategies (ES) are a class of black-box optimization algorithms and have been widely applied to solve problems, e.g., in reinforcement learning (RL), where the true gradient is unavailable. ES estimate the gradient of an objective function with respect to the parameters by randomly sampling search directions and evaluating parameter perturbations in these directions. However, the gradient estimator of ES tends to have a high variance for high-dimensional optimization, thus requiring a large number of samples and making ES inefficient. In this paper, we propose a new ES algorithm SGES, which utilizes historical estimated gradients to construct a low-dimensional subspace for sampling search directions, and adjusts the importance of this subspace adaptively. We prove that the variance of the gradient estimator of SGES can be much smaller than that of Vanilla ES; meanwhile, its bias can be well bounded. Empirical results on benchmark black-box functions and a set of popular RL tasks exhibit the superior performance of SGES over state-of-the-art ES algorithms.
Learning Adaptive Differential Evolution by Natural Evolution Strategies

Haotian Zhang,Jianyong Sun,Zongben Xu,Jialong Shi

DOI: https://doi.org/10.1109/tetci.2022.3210927

IF: 8.1

2023-01-01

Information Sciences

Abstract:Adaptive parameter control is critical in the design and application of evolutionary algorithm (EA), so does in differential evolution. In the past decade, many adaptive evolutionary algorithms have been proposed, in which online information collected until current generation during the evolutionary search procedure is used to determine the algorithmic parameters for the next generation. Recent studies often assume that the algorithmic parameters follow some distributions, while the distributions' parameters (called hyper-parameters) are updated by the collected information. Performances of these adaptive EAs depend highly on the hyper-parameters. Notice that the experiences obtained from optimizing some related problems could provide useful guidelines on how to adaptively control the distributions' parameters. However, few existing studies sufficiently used such experiences. To fill the gap, we propose a general framework for adaptive parameter control by modeling its evolution procedure as a Markov decision process. In the framework, a neural network is employed to act as the controller. The natural evolution strategies is applied to train the neural network. The proposed framework is applied on two well-known differential evolutions (DEs), namely JADE and LSHADE. By incorporating the learned controller, two DEs, named JADE/AC and LSHADE/AC, are formed. Experimental results on the CEC 2018 benchmark suite show that in general JADE/AC and LSHADE/AC perform significantly better than their counterparts. Moreover, in comparison with some well-known EAs including three suggested best DEs in a review paper (including LSHADE, cDE and CoBiDE), the championship algorithm in the CEC 2018 competitions, a recently-developed learnable DE and recently proposed DEs, our study shows that LSHADE/AC performs the best amongst them without sacrificing much computation time.
Evolution Strategies for Continuous Optimization: A Survey of the State-of-the-art

Zhenhua Li,Xi Lin,Qingfu Zhang,Hailin Liu

DOI: https://doi.org/10.1016/j.swevo.2020.100694

IF: 10.267

2020-01-01

Swarm and Evolutionary Computation

Abstract:Evolution strategies are a class of evolutionary algorithms for black-box optimization and achieve state-of-the-art performance on many benchmarks and real-world applications. Evolution strategies typically evolve a Gaussian distribution to approach the optimum. In this paper, we present a survey of recent advances in evolution strategies. We summarize the techniques, extensions, and practical considerations of evolution strategies for various optimization problems. We discuss some important open questions and promising topics that desire further research. Many of the discussed techniques and principles are applicable to other algorithms.
Evolution Strategies in Optimization Problems

Pedro A. F. Cruz,Delfim F. M. Torres

DOI: https://doi.org/10.48550/arXiv.0709.1020

2007-09-07

Abstract:Evolution Strategies are inspired in biology and part of a larger research field known as Evolutionary Algorithms. Those strategies perform a random search in the space of admissible functions, aiming to optimize some given objective function. We show that simple evolution strategies are a useful tool in optimal control, permitting to obtain, in an efficient way, good approximations to the solutions of some recent and challenging optimal control problems.

Optimization and Control
ES Is More Than Just a Traditional Finite-Difference Approximator

Joel Lehman,Jay Chen,Jeff Clune,Kenneth O. Stanley

DOI: https://doi.org/10.48550/arXiv.1712.06568

2018-05-02

Abstract:An evolution strategy (ES) variant based on a simplification of a natural evolution strategy recently attracted attention because it performs surprisingly well in challenging deep reinforcement learning domains. It searches for neural network parameters by generating perturbations to the current set of parameters, checking their performance, and moving in the aggregate direction of higher reward. Because it resembles a traditional finite-difference approximation of the reward gradient, it can naturally be confused with one. However, this ES optimizes for a different gradient than just reward: It optimizes for the average reward of the entire population, thereby seeking parameters that are robust to perturbation. This difference can channel ES into distinct areas of the search space relative to gradient descent, and also consequently to networks with distinct properties. This unique robustness-seeking property, and its consequences for optimization, are demonstrated in several domains. They include humanoid locomotion, where networks from policy gradient-based reinforcement learning are significantly less robust to parameter perturbation than ES-based policies solving the same task. While the implications of such robustness and robustness-seeking remain open to further study, this work's main contribution is to highlight such differences and their potential importance.

Neural and Evolutionary Computing,Artificial Intelligence
Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies

Oscar Li,James Harrison,Jascha Sohl-Dickstein,Virginia Smith,Luke Metz

2023-12-10

Abstract:Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivtiy, discontinuity, or blackbox characteristics. In such scenarios, online evolution strategies methods are a more capable alternative, while being more parallelizable than vanilla evolution strategies (ES) by interleaving partial unrolls and gradient updates. In this work, we propose a general class of unbiased online evolution strategies methods. We analytically and empirically characterize the variance of this class of gradient estimators and identify the one with the least variance, which we term Noise-Reuse Evolution Strategies (NRES). Experimentally, we show NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.

Neural and Evolutionary Computing,Artificial Intelligence,Machine Learning
NGDE: A Niching-Based Gradient-Directed Evolution Algorithm for Nonconvex Optimization

Qi Yu,Xijun Liang,Mengzhen Li,Ling Jian

DOI: https://doi.org/10.1109/TNNLS.2024.3378805

2024-04-15

Abstract:Nonconvex optimization issues are prevalent in machine learning and data science. While gradient-based optimization algorithms can rapidly converge and are dimension-independent, they may, unfortunately, fall into local optimal solutions or saddle points. In contrast, evolutionary algorithms (EAs) gradually adapt the population of solutions to explore global optimal solutions. However, this approach requires substantial computational resources to perform numerous fitness function evaluations, which poses challenges for high-dimensional optimization in particular. This study introduces a novel nonconvex optimization algorithm, the niching-based gradient-directed evolution (NGDE) algorithm, designed specifically for high-dimensional nonconvex optimization. The NGDE algorithm generates potential solutions and divides them into multiple niches to explore distinct areas within the feasible region. Subsequently, each individual creates candidate offspring using the gradient-directed mutation operator we designed. The convergence properties of the NGDE algorithm are investigated in two scenarios: accessing the full gradient and approximating the gradient with mini-batch samples. The experimental studies demonstrate the superior performance of the NGDE algorithm in minimizing multimodal optimization functions. Additionally, when applied to train the neural networks of LeNet-5, NGDE shows significantly improved classification accuracy, especially in smaller training sizes.
A New Multi-Objective Evolutionary Algorithm: Neighbourhood Exploring Evolution Strategy

XL Hu,CAC Coello,ZC Huang

DOI: https://doi.org/10.1080/03052150500035658

IF: 2.5

2005-01-01

Engineering Optimization

Abstract:This article proposes a new multi-objective evolutionary algorithm, called neighbourhood exploring evolution strategy (NEES). This approach incorporates the idea of neighbourhood exploration together with other techniques commonly used in the multi-objective evolutionary optimization literature (namely, non-dominated sorting and diversity preservation mechanisms). The main idea of the proposed approach was derived from a single-objective evolutionary algorithm, called the line-up competition algorithm (LCA), and it consists of assigning neighbourhoods of different sizes to different solutions. Within each neighbourhood, new solutions are generated using a (1+λ)-ES (evolution strategy). This scheme naturally balances the effect of local search (which is performed by the neighbourhood exploration mechanism) with that of the global search performed by the algorithm, and gradually impels the population to progress towards the true Pareto-optimal front of the problem to explore the extent of that front. Three versions of the proposal are studied: a (1+1)-NEES, a (1+2)-NEES and a (1+4)-NEES. Such approaches are validated on a set of standard test problems reported in the specialized literature. Simulation results indicate that, for continuous numerical optimization problems, the proposal (particularly the (1+1)-NEES) is competitive with respect to NSGA-II, which is an algorithm representative of the state-of-the-art in evolutionary multi-objective optimization. Moreover, all the versions of NEES improve on the results of NSGA-II when dealing with a discrete optimization problem. Although preliminary, such results might indicate a potential application area in which the proposed approach could be particularly useful.
Guided evolutionary strategies: Augmenting random search with surrogate gradients

Niru Maheswaranathan,Luke Metz,George Tucker,Dami Choi,Jascha Sohl-Dickstein

DOI: https://doi.org/10.48550/arXiv.1806.10230

2019-06-11

Abstract:Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute than the full gradient (e.g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e.g. in certain reinforcement learning applications, or when using synthetic gradients). We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search. We define a search distribution for evolutionary strategies that is elongated along a guiding subspace spanned by the surrogate gradients. This allows us to estimate a descent direction which can then be passed to a first-order optimizer. We analytically and numerically characterize the tradeoffs that result from tuning how strongly the search distribution is stretched along the guiding subspace, and we use this to derive a setting of the hyperparameters that works well across problems. Finally, we apply our method to example problems, demonstrating an improvement over both standard evolutionary strategies and first-order methods (that directly follow the surrogate gradient). We provide a demo of Guided ES at <a class="link-external link-https" href="https://github.com/brain-research/guided-evolutionary-strategies" rel="external noopener nofollow">this https URL</a>

Neural and Evolutionary Computing,Machine Learning
Information-Geometric Optimization with Natural Selection

Jakub Otwinowski,Colin H LaMont,Armita Nourmohammad

DOI: https://doi.org/10.3390/e22090967

2020-08-31

Abstract:Evolutionary algorithms, inspired by natural evolution, aim to optimize difficult objective functions without computing derivatives. Here we detail the relationship between classical population genetics of quantitative traits and evolutionary optimization, and formulate a new evolutionary algorithm. Optimization of a continuous objective function is analogous to searching for high fitness phenotypes on a fitness landscape. We describe how natural selection moves a population along the non-Euclidean gradient that is induced by the population on the fitness landscape (the natural gradient). We show how selection is related to Newton's method in optimization under quadratic fitness landscapes, and how selection increases fitness at the cost of reducing diversity. We describe the generation of new phenotypes and introduce an operator that recombines the whole population to generate variants. Finally, we introduce a proof-of-principle algorithm that combines natural selection, our recombination operator, and an adaptive method to increase selection and find the optimum. The algorithm is extremely simple in implementation; it has no matrix inversion or factorization, does not require storing a covariance matrix, and may form the basis of more general model-based optimization algorithms with natural gradient updates.
Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions

Florian Meier,Asier Mujika,Marcelo Matheus Gauy,Angelika Steger

DOI: https://doi.org/10.48550/arXiv.1910.05268

2019-10-12

Abstract:Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for deep neural networks when the true gradients cannot be computed, such as in Reinforcement Learning. We continue a recent line of research that uses surrogate gradients to improve the gradient estimation of ES. We propose a novel method to optimally incorporate surrogate gradient information. Our approach, unlike previous work, needs no information about the quality of the surrogate gradients and is always guaranteed to find a descent direction that is better than the surrogate gradient. This allows to iteratively use the previous gradient estimate as surrogate gradient for the current search point. We theoretically prove that this yields fast convergence to the true gradient for linear functions and show under simplifying assumptions that it significantly improves gradient estimates for general functions. Finally, we evaluate our approach empirically on MNIST and reinforcement learning tasks and show that it considerably improves the gradient estimation of ES at no extra computational cost.

Neural and Evolutionary Computing,Machine Learning
A knowledge transfer-based adaptive differential evolution for solving nonlinear equation systems

Zuowen Liao,Qiong Gu,Shuijia Li,Yu Sun

DOI: https://doi.org/10.1016/j.knosys.2022.110214

2022-12-29

Abstract:Solving nonlinear equation systems (NESs) is an important yet challenging task in the field of numerical computation. It aims to locate multiple roots in a single run. However, the existing methods lack effective knowledge transfer. In this article, a knowledge transfer-based adaptive differential evolution is proposed to deal with NESs. Its main features are: (i) knowledge transfer between two niching techniques (crowding and speciation) is carried out to balance diversity and convergence; (ii) the variation characteristics of population diversity and convergence are used to judge knowledge transfer intensity; (iii) a knowledge transfer mechanism is designed to ensure that reasonable individuals are selected for the transfer to supplement the deficiencies of crowding and speciation; (iv) a parameter adaptation with niching level is introduced to improve search efficiency. Experiments on classical 30 NES problems have demonstrated that the proposed approach can outperform the state-of-the-art algorithms, in terms of root ratio and success rate.

computer science, artificial intelligence
A differential evolution algorithm for solving mixed-integer nonlinear programming problems

Daniel Molina-Pérez,Efrén Mezura-Montes,Edgar Alfredo Portilla-Flores,Eduardo Vega-Alvarado,Bárbara Calva-Yañez

DOI: https://doi.org/10.1016/j.swevo.2023.101427

IF: 10.267

2024-02-01

Swarm and Evolutionary Computation

Abstract:Many engineering optimization problems fall into the category of Mixed-Integer Nonlinear Programming (MINLP) problems, which combine nonlinear relations, constraint conditions, and different types of variables, including continuous, integer, and/or discrete variables. Solving MINLP problems can be a challenging exploration process since their landscape can be composed of many discontinuous feasible parts with different sizes. In this scheme, Evolutionary Algorithms (EAs) often fail to generate enough diversity to explore the discontinuous feasible parts. Consequently, EAs are vulnerable to being attracted to larger discontinuous feasible parts, even if they are not promising regions. In this paper, a new proposal based on two fundamental strategies is presented to improve the performance of the differential evolution algorithm when solving MINLP problems. The first strategy considers a set of “good fitness-infeasible solutions” that contribute to exploring promising regions from infeasible contours. It reduces the vulnerability of the solutions to be attracted to larger discontinuous feasible parts with unpromising objective function values. The second is a composite trial vector generation to improve the combinatorial exploration while ensuring a robust convergence capacity toward the final solution. Sixteen well-known MINLP problems are used in several experiments to evaluate the performance of the proposed algorithm, comparing it to state-of-the-art EAs. The results provided by the proposal show a better performance in terms of quality, robustness, and computational cost.

computer science, artificial intelligence, theory & methods
Challenges in High-Dimensional Reinforcement Learning with Evolution Strategies

Nils Müller,Tobias Glasmachers

DOI: https://doi.org/10.1007/978-3-319-99259-4_33

2018-01-01

Abstract:Evolution Strategies (ESs) have recently become popular for training deep neural networks, in particular on reinforcement learning tasks, a special form of controller design. Compared to classic problems in continuous direct search, deep networks pose extremely high-dimensional optimization problems, with many thousands or even millions of variables. In addition, many control problems give rise to a stochastic fitness function. Considering the relevance of the application, we study the suitability of evolution strategies for high-dimensional, stochastic problems. Our results give insights into which algorithmic mechanisms of modern ES are of value for the class of problems at hand, and they reveal principled limitations of the approach. They are in line with our theoretical understanding of ESs. We show that combining ESs that offer reduced internal algorithm cost with uncertainty handling techniques yields promising methods for this class of problems.

Discrete Natural Evolution Strategies

Natural Evolution Strategy for Mixed-Integer Black-Box Optimization

Fast Moving Natural Evolution Strategy for High-Dimensional Problems

Look-ahead natural evolutionary strategies

Hard-Thresholding Meets Evolution Strategies in Reinforcement Learning

Generative Evolutionary Strategy For Black-Box Optimizations

Maximum Entropy Reinforcement Learning with Evolution Strategies

Self-guided Evolution Strategies with Historical Estimated Gradients

Learning Adaptive Differential Evolution by Natural Evolution Strategies

Evolution Strategies for Continuous Optimization: A Survey of the State-of-the-art

Evolution Strategies in Optimization Problems

ES Is More Than Just a Traditional Finite-Difference Approximator

Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies

NGDE: A Niching-Based Gradient-Directed Evolution Algorithm for Nonconvex Optimization

A New Multi-Objective Evolutionary Algorithm: Neighbourhood Exploring Evolution Strategy

Guided evolutionary strategies: Augmenting random search with surrogate gradients

Information-Geometric Optimization with Natural Selection

Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions

A knowledge transfer-based adaptive differential evolution for solving nonlinear equation systems

A differential evolution algorithm for solving mixed-integer nonlinear programming problems

Challenges in High-Dimensional Reinforcement Learning with Evolution Strategies