Abstract:In this paper, we introduce a new heuristics for global optimization in scenarios where extensive evaluations of the cost function are expensive, inaccessible, or even prohibitive. The method, which we call Landscape-Sketch-and-Step (LSS), combines Machine Learning, Stochastic Optimization, and Reinforcement Learning techniques, relying on historical information from previously sampled points to make judicious choices of parameter values where the cost function should be evaluated at. Unlike optimization by Replica Exchange Monte Carlo methods, the number of evaluations of the cost function required in this approach is comparable to that used by Simulated Annealing, quality that is especially important in contexts like high-throughput computing or high-performance computing tasks, where evaluations are either computationally expensive or take a long time to be performed. The method also differs from standard Surrogate Optimization techniques, for it does not construct a surrogate model that aims at approximating or reconstructing the objective function. We illustrate our method by applying it to low dimensional optimization problems (dimensions 1, 2, 4, and 8) that mimick known difficulties of minimization on rugged energy landscapes often seen in Condensed Matter Physics, where cost functions are rugged and plagued with local minima. When compared to classical Simulated Annealing, the LSS shows an effective acceleration of the optimization process.

When Does MAML Objective Have Benign Landscape?

How Does the Task Landscape Affect MAML Performance?

On the Global Optimality of Model-Agnostic Meta-Learning

Meta Learning in the Continuous Time Limit

When MAML Can Adapt Fast and How to Assist When It Cannot

Convergence of Gradient-based MAML in LQR

When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications

Alpha MAML: Adaptive Model-Agnostic Meta-Learning

A New First-Order Meta-Learning Algorithm with Convergence Guarantees

On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms

A Landscape-Aware Differential Evolution for Multimodal Optimization Problems

Landscape-Sketch-Step: An AI/ML-Based Metaheuristic for Surrogate Optimization Problems

Model-Agnostic Meta-Learning for Natural Language Understanding Tasks in Finance

MAML is a Noisy Contrastive Learner in Classification

A Model-Agnostic Approach to Mitigate Gradient Interference for Multi-Task Learning

Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD

Many-objective ant lion optimizer (MaOALO): A new many-objective optimizer with its engineering applications

Neural Exploratory Landscape Analysis

Bi-Objective Lexicographic Optimization in Markov Decision Processes with Related Objectives

Independent Component Alignment for Multi-Task Learning

Meta-Learning with a Geometry-Adaptive Preconditioner