Accelerated Optimization for Machine Learning

Zhouchen Lin,Huan Li,Cong Fang

DOI: https://doi.org/10.1007/978-981-15-2910-8

2020-01-01

Abstract:This is the first monograph that reviews the state-of-the-art accelerated first-order optimization algorithms used in machine learning. This book is comprehensive, up-to-date and self-contained, easy for beginners to grasp the frontiers of optimization in machine learning.

What problem does this paper attempt to address?

Accelerated First-Order Optimization Algorithms for Machine Learning.

Huan Li,Cong Fang,Zhouchen Lin

DOI: https://doi.org/10.1109/jproc.2020.3007634

IF: 20.6

2020-01-01

Proceedings of the IEEE

Abstract:Numerical optimization serves as one of the pillars of machine learning. To meet the demands of big data applications, lots of efforts have been put on designing theoretically and practically fast algorithms. This article provides a comprehensive survey on accelerated first-order algorithms with a focus on stochastic algorithms. Specifically, this article starts with reviewing the basic accelerated algorithms on deterministic convex optimization, then concentrates on their extensions to stochastic convex optimization, and at last introduces some recent developments on acceleration for nonconvex optimization.
First-order and Stochastic Optimization Methods for Machine Learning

Guanghui Lan

DOI: https://doi.org/10.1007/978-3-030-39568-1

2020-01-01

Abstract:This book covers not only foundational materials but also the most recent progresses made during the past few years on the area of machine learning algorithms. In spite of the intensive research and development in this area, there does not exist a systematic treatment to introduce the fundamental concepts and recent progresses on machine learning algorithms, especially on those based on stochastic optimization methods, randomized algorithms, nonconvex optimization, distributed and online learning, and projection free methods. This book will benefit the broad audience in the area of machine learning, artificial intelligence and mathematical programming community by presenting these recent developments in a tutorial style, starting from the basic building blocks to the most carefully designed and complicated algorithms for machine learning.
Black Box Optimization, Machine Learning, and No-Free Lunch Theorems

DOI: https://doi.org/10.1007/978-3-030-66515-9

2021-01-01

Abstract:This edited volume illustrates the connections between machine learning techniques, black box optimization, and no-free lunch theorems. Each of the thirteen contributions focuses on the commonality and interdisciplinary concepts as well as the fundamentals needed to fully comprehend the impact of individual applications and problems. Current theoretical, algorithmic, and practical methods used are provided to stimulate a new effort towards innovative and efficient solutions. The book is intended for beginners who wish to achieve a broad overview of optimization methods and also for more experienced researchers as well as researchers in mathematics, optimization, operations research, quantitative logistics, data analysis, and statistics, who will benefit from access to a quick reference to key topics and methods. The coverage ranges from mathematically rigorous methods to heuristic and evolutionary approaches in an attempt to equip the reader with different viewpoints of the same problem.
Accelerated Algorithms for Nonconvex Optimization

Zhouchen Lin,Huan Li,Cong Fang

DOI: https://doi.org/10.1007/978-981-15-2910-8_4

2020-01-01

Abstract:This chapter reviews the representative accelerated algorithms for deterministic unconstrained nonconvex optimization. We start with introducing the proximal gradient method with momentum for general nonconvex problems, including its convergence to the critical points and convergence rates under the Kurdyka–Łojasiewicz condition. Then, we study the provable faster convergence of the accelerated gradient method to the critical points, which is equipped with negative curvature exploitation. At last, we further introduce the accelerated algorithms with the faster convergence guarantee to the approximate local optima, which escape from the saddle points quickly.
A Survey of Optimization Methods From a Machine Learning Perspective

Shiliang Sun,Zehui Cao,Han Zhu,Jing Zhao

DOI: https://doi.org/10.1109/tcyb.2019.2950779

IF: 11.8

2020-08-01

IEEE Transactions on Cybernetics

Abstract:Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this article, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Finally, we explore and give some challenges and open problems for the optimization in machine learning.

automation & control systems,computer science, cybernetics, artificial intelligence
Accelerated First-Order Optimization under Nonlinear Constraints

Michael Muehlebach,Michael I. Jordan

2024-01-02

Abstract:We exploit analogies between first-order algorithms for constrained optimization and non-smooth dynamical systems to design a new class of accelerated first-order algorithms for constrained optimization. Unlike Frank-Wolfe or projected gradients, these algorithms avoid optimization over the entire feasible set at each iteration. We prove convergence to stationary points even in a nonconvex setting and we derive accelerated rates for the convex setting both in continuous time, as well as in discrete time. An important property of these algorithms is that constraints are expressed in terms of velocities instead of positions, which naturally leads to sparse, local and convex approximations of the feasible set (even if the feasible set is nonconvex). Thus, the complexity tends to grow mildly in the number of decision variables and in the number of constraints, which makes the algorithms suitable for machine learning applications. We apply our algorithms to a compressed sensing and a sparse regression problem, showing that we can treat nonconvex $\ell^p$ constraints ($p<1$) efficiently, while recovering state-of-the-art performance for $p=1$.

Optimization and Control,Machine Learning,Signal Processing
A Comparison of First-order Algorithms for Machine Learning

Yu Wei,Pock Thomas

DOI: https://doi.org/10.48550/arXiv.1404.6674

2014-04-27

Abstract:Using an optimization algorithm to solve a machine learning problem is one of mainstreams in the field of science. In this work, we demonstrate a comprehensive comparison of some state-of-the-art first-order optimization algorithms for convex optimization problems in machine learning. We concentrate on several smooth and non-smooth machine learning problems with a loss function plus a regularizer. The overall experimental results show the superiority of primal-dual algorithms in solving a machine learning problem from the perspectives of the ease to construct, running time and accuracy.

Machine Learning
Accelerated Algorithms for Constrained Convex Optimization

Zhouchen Lin,Huan Li,Cong Fang

DOI: https://doi.org/10.1007/978-981-15-2910-8_3

2020-01-01

Abstract:This chapter reviews the representative accelerated algorithms for deterministic constrained convex optimization. We overview the accelerated penalty method, accelerated Lagrange multiplier method, and the accelerated augmented Lagrange multiplier method. In particular, we concentrate on two widely used algorithms, namely the alternating direction method of multiplier (ADMM) and the primal-dual method. For ADMM, we study four scenarios, namely the generally convex and nonsmooth case, the strongly convex and nonsmooth case, the generally convex and smooth case, and the strongly convex and smooth case. We also introduce its non-ergodic accelerated variant. For the primal-dual method, we study three scenarios: both the two functions are generally convex, both are strongly convex, and one is generally convex, while the other is strongly convex. Finally, we introduce the Frank–Wolfe algorithm …
Optimization Methods in Deep Learning: A Comprehensive Overview

David Shulman

DOI: https://doi.org/10.48550/arXiv.2302.09566

2023-04-24

Abstract:In recent years, deep learning has achieved remarkable success in various fields such as image recognition, natural language processing, and speech recognition. The effectiveness of deep learning largely depends on the optimization methods used to train deep neural networks. In this paper, we provide an overview of first-order optimization methods such as Stochastic Gradient Descent, Adagrad, Adadelta, and RMSprop, as well as recent momentum-based and adaptive gradient methods such as Nesterov accelerated gradient, Adam, Nadam, AdaMax, and AMSGrad. We also discuss the challenges associated with optimization in deep learning and explore techniques for addressing these challenges, including weight initialization, batch normalization, and layer normalization. Finally, we provide recommendations for selecting optimization methods for different deep learning tasks and datasets. This paper serves as a comprehensive guide to optimization methods in deep learning and can be used as a reference for researchers and practitioners in the field.

Machine Learning,Optimization and Control
Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

Yunlong Yu,Fuxian Liu

DOI: https://doi.org/10.1109/access.2019.2919987

IF: 3.9

2019-01-01

IEEE Access

Abstract:First-order gradient-based optimization algorithms have been of core practical importance in the field of deep learning. In this paper, we propose a new weighting mechanism-based first-order gradient descent optimization algorithm, namely NWM-Adam, to resolve the undesirable convergence behavior of some optimization algorithms which employ fixed sized window of past gradients to scale the gradient updates and improve the performance of Adam and AMSGrad. The NWM-Adam is developed on the basis of the idea, i.e., placing more memory of the past gradients than the recent gradients. Furthermore, it can easily adjust the degree to which how much the past gradients weigh in the estimation. In order to empirically test the performance of our proposed NWM-Adam optimization algorithm, we compare it with other popular optimization algorithms in three well-known machine learning models, i.e., logistic regression, multi-layer fully connected neural networks, and deep convolutional neural networks. The experimental results show that the NWM-Adam can outperform other optimization algorithms.
Non-convex Optimization for Machine Learning

Prateek Jain,Purushottam Kar

DOI: https://doi.org/10.1561/2200000058

2017-01-01

Foundations and Trends® in Machine Learning

Abstract:A vast majority of machine learning algorithms train their models and perform inference by solving optimization problems. In order to capture the learning and prediction problems accurately, structural constraints such as sparsity or low rank are frequently imposed or else the objective itself is designed to be a non-convex function. This is especially true of algorithms that operate in high-dimensional spaces or that train non-linear models such as tensor models and deep networks. The freedom to express the learning problem as a non-convex optimization problem gives immense modeling power to the algorithm designer, but often such problems are NP-hard to solve. A popular workaround to this has been to relax non-convex problems to convex ones and use traditional methods to solve the (convex) relaxed optimization problems. However this approach may be lossy and nevertheless presents significant challenges for large scale optimization. On the other hand, direct approaches to non-convex optimization have met with resounding success in several domains and remain the methods of choice for the practitioner, as they frequently outperform relaxation-based techniques – popular heuristics include projected gradient descent and alternating minimization. However, these are often poorly understood in terms of their convergence and other properties. This monograph presents a selection of recent advances that bridge a long-standing gap in our understanding of these heuristics. We hope that an insight into the inner workings of these methods will allow the reader to appreciate the unique marriage of task structure and generative models that allow these heuristic techniques to (provably) succeed. The monograph will lead the reader through several widely used nonconvex optimization techniques, as well as applications thereof. The goal of this monograph is to both, introduce the rich literature in this area, as well as equip the reader with the tools and techniques needed to analyze these simple procedures for non-convex problems.
Optimization for deep learning: theory and algorithms

Ruoyu Sun

DOI: https://doi.org/10.48550/arXiv.1912.08957

2019-12-19

Abstract:When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including careful initialization and normalization methods. Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms. Third, we review existing research on the global issues of neural network training, including results on bad local minima, mode connectivity, lottery ticket hypothesis and infinite-width analysis.

Machine Learning,Optimization and Control
Adaptive Optimization Algorithms for Machine Learning

Slavomír Hanzely

2023-11-17

Abstract:Machine learning assumes a pivotal role in our data-driven world. The increasing scale of models and datasets necessitates quick and reliable algorithms for model training. This dissertation investigates adaptivity in machine learning optimizers. The ensuing chapters are dedicated to various facets of adaptivity, including: 1. personalization and user-specific models via personalized loss, 2. provable post-training model adaptations via meta-learning, 3. learning unknown hyperparameters in real time via hyperparameter variance reduction, 4. fast O(1/k^2) global convergence of second-order methods via stepsized Newton method regardless of the initialization and choice basis, 5. fast and scalable second-order methods via low-dimensional updates. This thesis contributes novel insights, introduces new algorithms with improved convergence guarantees, and improves analyses of popular practical algorithms.

Machine Learning,Optimization and Control
Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Yuhan Ma,Dan Sun,Erdi Gao,Ningjing Sang,Iris Li,Guanming Huang

DOI: https://doi.org/10.48550/arXiv.2409.04707

2024-09-07

Abstract:Optimization theory serves as a pivotal scientific instrument for achieving optimal system performance, with its origins in economic applications to identify the best investment strategies for maximizing benefits. Over the centuries, from the geometric inquiries of ancient Greece to the calculus contributions by Newton and Leibniz, optimization theory has significantly advanced. The persistent work of scientists like Lagrange, Cauchy, and von Neumann has fortified its progress. The modern era has seen an unprecedented expansion of optimization theory applications, particularly with the growth of computer science, enabling more sophisticated computational practices and widespread utilization across engineering, decision analysis, and operations research. This paper delves into the profound relationship between optimization theory and deep learning, highlighting the omnipresence of optimization problems in the latter. We explore the gradient descent algorithm and its variants, which are the cornerstone of optimizing neural networks. The chapter introduces an enhancement to the SGD optimizer, drawing inspiration from numerical optimization methods, aiming to enhance interpretability and accuracy. Our experiments on diverse deep learning tasks substantiate the improved algorithm's efficacy. The paper concludes by emphasizing the continuous development of optimization theory and its expanding role in solving intricate problems, enhancing computational capabilities, and informing better policy decisions.

Machine Learning,Artificial Intelligence
Optimization with First Order Algorithms

Charles Dossal,Samuel Hurault,Nicolas Papadakis

2024-10-25

Abstract:These notes focus on the minimization of convex functionals using first-order optimization methods, which are fundamental in many areas of applied mathematics and engineering. The primary goal of this document is to introduce and analyze the most classical first-order optimization algorithms. We aim to provide readers with both a practical and theoretical understanding in how and why these algorithms converge to minimizers of convex functions. The main algorithms covered in these notes include gradient descent, Forward-Backward splitting, Douglas-Rachford splitting, the Alternating Direction Method of Multipliers (ADMM), and Primal-Dual algorithms. All these algorithms fall into the class of first order methods, as they only involve gradients and subdifferentials, that are first order derivatives of the functions to optimize. For each method, we provide convergence theorems, with precise assumptions and conditions under which the convergence holds, accompanied by complete proofs. Beyond convex optimization, the final part of this manuscript extends the analysis to nonconvex problems, where we discuss the convergence behavior of these same first-order methods under broader assumptions. To contextualize the theory, we also include a selection of practical examples illustrating how these algorithms are applied in different image processing problems.

Optimization and Control
Machine Learning Optimization Techniques: A Survey, Classification, Challenges, and Future Research Issues

Bian, Kewei,Priyadarshi, Rahul

DOI: https://doi.org/10.1007/s11831-024-10110-w

IF: 9.7

2024-03-30

Archives of Computational Methods in Engineering

Abstract:Optimization approaches in machine learning (ML) are essential for training models to obtain high performance across numerous domains. The article provides a comprehensive overview of ML optimization strategies, emphasizing their classification, obstacles, and potential areas for further study. We proceed with studying the historical progression of optimization methods, emphasizing significant developments and their influence on contemporary algorithms. We analyse the present research to identify widespread optimization algorithms and their uses in supervised learning, unsupervised learning, and reinforcement learning. Various common optimization constraints, including non-convexity, scalability issues, convergence problems, and concerns about robustness and generalization, are also explored. We suggest future research should focus on scalability problems, innovative optimization techniques, domain knowledge integration, and improving interpretability. The present study aims to provide an in-depth review of ML optimization by combining insights from historical advancements, literature evaluations, and current issues to guide future research efforts.

computer science, interdisciplinary applications,engineering, multidisciplinary,mathematics
Accelerated Forward-Backward Optimization Using Deep Learning

Sebastian Banert,Jevgenija Rudzusika,Ozan Öktem,Jonas Adler

DOI: https://doi.org/10.1137/22m1532548

IF: 2.763

2024-04-03

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1236-1263, June 2024. We propose several deep-learning accelerated optimization solvers with convergence guarantees. We use ideas from the analysis of accelerated forward-backward schemes like FISTA, but instead of the classical approach of proving convergence for a choice of parameters, such as a step-size, we show convergence whenever the update is chosen in a specific set. Rather than picking a point in this set using some predefined method, we train a deep neural network to pick the best update within a given space. Finally, we show that the method is applicable to several cases of smooth and nonsmooth optimization and show superior results to established accelerated solvers.

mathematics, applied
Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Yuanshi Liu,Hanzhen Zhao,Yang Xu,Pengyun Yue,Cong Fang

2023-12-06

Abstract:Gradient-based minimax optimal algorithms have greatly promoted the development of continuous optimization and machine learning. One seminal work due to Yurii Nesterov [Nes83a] established $\tilde{\mathcal{O}}(\sqrt{L/\mu})$ gradient complexity for minimizing an $L$-smooth $\mu$-strongly convex objective. However, an ideal algorithm would adapt to the explicit complexity of a particular objective function and incur faster rates for simpler problems, triggering our reconsideration of two defeats of existing optimization modeling and analysis. (i) The worst-case optimality is neither the instance optimality nor such one in reality. (ii) Traditional $L$-smoothness condition may not be the primary abstraction/characterization for modern practical problems. In this paper, we open up a new way to design and analyze gradient-based algorithms with direct applications in machine learning, including linear regression and beyond. We introduce two factors $(\alpha, \tau_{\alpha})$ to refine the description of the degenerated condition of the optimization problems based on the observation that the singular values of Hessian often drop sharply. We design adaptive algorithms that solve simpler problems without pre-known knowledge with reduced gradient or analogous oracle accesses. The algorithms also improve the state-of-art complexities for several problems in machine learning, thereby solving the open problem of how to design faster algorithms in light of the known complexity lower bounds. Specially, with the $\mathcal{O}(1)$-nuclear norm bounded, we achieve an optimal $\tilde{\mathcal{O}}(\mu^{-1/3})$ (v.s. $\tilde{\mathcal{O}}(\mu^{-1/2})$) gradient complexity for linear regression. We hope this work could invoke the rethinking for understanding the difficulty of modern problems in optimization.

Machine Learning,Computational Complexity
Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

Zhenwei Lin,Qi Deng

2024-11-27

Abstract:In this paper, we introduce faster accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex function constraints. Prior to our work, the best complexity bound was $\mathcal{O}(1/{\varepsilon})$, regardless of the strong convexity of the constraint function. It is unclear whether the strong convexity assumption can enable even better convergence results. To address this issue, we have developed novel techniques to progressively estimate the strong convexity of the Lagrangian function. Our approach, for the first time, effectively leverages the constraint strong convexity, obtaining an improved complexity of $\mathcal{O}(1/\sqrt{\varepsilon})$. This rate matches the complexity lower bound for strongly-convex-concave saddle point optimization and is therefore order-optimal. We show the superior performance of our methods in sparsity-inducing constrained optimization, notably Google's personalized PageRank problem. Furthermore, we show that a restarted version of the proposed methods can effectively identify the optimal solution's sparsity pattern within a finite number of steps, a result that appears to have independent significance.

Optimization and Control,Machine Learning

Accelerated Optimization for Machine Learning

Accelerated First-Order Optimization Algorithms for Machine Learning.

First-order and Stochastic Optimization Methods for Machine Learning

Black Box Optimization, Machine Learning, and No-Free Lunch Theorems

Accelerated Algorithms for Nonconvex Optimization

A Survey of Optimization Methods From a Machine Learning Perspective

Accelerated First-Order Optimization under Nonlinear Constraints

A Comparison of First-order Algorithms for Machine Learning

Accelerated Algorithms for Constrained Convex Optimization

Optimization Methods in Deep Learning: A Comprehensive Overview

Effective Neural Network Training with a New Weighting Mechanism-Based Optimization Algorithm.

Non-convex Optimization for Machine Learning

Optimization for deep learning: theory and algorithms

Adaptive Optimization Algorithms for Machine Learning

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Optimization with First Order Algorithms

Machine Learning Optimization Techniques: A Survey, Classification, Challenges, and Future Research Issues

Accelerated Forward-Backward Optimization Using Deep Learning

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints