Lecture Notes: Optimization for Machine Learning

Elad Hazan

DOI: https://doi.org/10.48550/arXiv.1909.03550

IF: 5.414

2019-09-08

Machine Learning

Abstract:Lecture notes on optimization for machine learning, derived from a course at Princeton University and tutorials given in MLSS, Buenos Aires, as well as Simons Foundation, Berkeley.

What problem does this paper attempt to address?

Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning

Nadav Cohen,Noam Razin

2024-08-25

Abstract:These notes are based on a lecture delivered by NC on March 2021, as part of an advanced course in Princeton University on the mathematical understanding of deep learning. They present a theory (developed by NC, NR and collaborators) of linear neural networks -- a fundamental model in the study of optimization and generalization in deep learning. Practical applications born from the presented theory are also discussed. The theory is based on mathematical tools that are dynamical in nature. It showcases the potential of such tools to push the envelope of our understanding of optimization and generalization in deep learning. The text assumes familiarity with the basics of statistical learning theory. Exercises (without solutions) are included.

Machine Learning,Artificial Intelligence
A Course in Dynamic Optimization

Bar Light

2024-10-10

Abstract:These lecture notes are derived from a graduate-level course in dynamic optimization, offering an introduction to techniques and models extensively used in management science, economics, operations research, engineering, and computer science. The course emphasizes the theoretical underpinnings of discrete-time dynamic programming models and advanced algorithmic strategies for solving these models. Unlike typical treatments, it provides a proof for the principle of optimality for upper semi-continuous dynamic programming, a middle ground between the simpler countable state space case \cite{bertsekas2012dynamic}, and the involved universally measurable case \cite{bertsekas1996stochastic}. This approach is sufficiently rigorous to include important examples such as dynamic pricing, consumption-savings, and inventory management models. The course also delves into the properties of value and policy functions, leveraging classical results \cite{topkis1998supermodularity} and recent developments. Additionally, it offers an introduction to reinforcement learning, including a formal proof of the convergence of Q-learning algorithms. Furthermore, the notes delve into policy gradient methods for the average reward case, presenting a convergence result for the tabular case in this context. This result is simple and similar to the discounted case but appears to be new.

Optimization and Control,Theoretical Economics,Systems and Control
Lecture Notes on Free Probability

Vladislav Kargin

DOI: https://doi.org/10.48550/arXiv.1305.2611

2013-05-13

Abstract:Lecture notes for a graduate course lectured at Stanford University.

Probability
Introduction to Machine Learning: Class Notes 67577

Amnon Shashua

DOI: https://doi.org/10.48550/arXiv.0904.3664

IF: 5.414

2009-04-23

Machine Learning

Abstract:Introduction to Machine learning covering Statistical Inference (Bayes, EM, ML/MaxEnt duality), algebraic and spectral methods (PCA, LDA, CCA, Clustering), and PAC learning (the Formal model, VC dimension, Double Sampling theorem).
Introduction to Machine Learning for the Sciences

Titus Neupert,Mark H Fischer,Eliska Greplova,Kenny Choo,M. Michael Denner

DOI: https://doi.org/10.48550/arXiv.2102.04883

2022-06-22

Abstract:This is an introductory machine-learning course specifically developed with STEM students in mind. Our goal is to provide the interested reader with the basics to employ machine learning in their own projects and to familiarize themself with the terminology as a foundation for further reading of the relevant literature. In these lecture notes, we discuss supervised, unsupervised, and reinforcement learning. The notes start with an exposition of machine learning methods without neural networks, such as principle component analysis, t-SNE, clustering, as well as linear regression and linear classifiers. We continue with an introduction to both basic and advanced neural-network structures such as dense feed-forward and conventional neural networks, recurrent neural networks, restricted Boltzmann machines, (variational) autoencoders, generative adversarial networks. Questions of interpretability are discussed for latent-space representations and using the examples of dreaming and adversarial attacks. The final section is dedicated to reinforcement learning, where we introduce basic notions of value functions and policy learning.

Computational Physics,Disordered Systems and Neural Networks,Machine Learning
Lecture Notes: Neural Network Architectures

Evelyn Herberg

DOI: https://doi.org/10.48550/arXiv.2304.05133

IF: 5.414

2023-04-11

Machine Learning

Abstract:These lecture notes provide an overview of Neural Network architectures from a mathematical point of view. Especially, Machine Learning with Neural Networks is seen as an optimization problem. Covered are an introduction to Neural Networks and the following architectures: Feedforward Neural Network, Convolutional Neural Network, ResNet, and Recurrent Neural Network.
Quantum algorithms for optimizers

Giacomo Nannicini

2024-09-17

Abstract:This is a set of lecture notes for a Ph.D.-level course on quantum algorithms, with an emphasis on quantum optimization algorithms. It is developed for applied mathematicians and engineers, and requires no previous background in quantum mechanics. The main topics of this course, in addition to a rigorous introduction to the computational model, are: input/output models, quantum search, the quantum gradient algorithm, matrix manipulation algorithms, the matrix multiplicative weights update framework for semidefinite optimization, adiabatic optimization.

Quantum Physics,Discrete Mathematics,Optimization and Control
Black Box Optimization, Machine Learning, and No-Free Lunch Theorems

DOI: https://doi.org/10.1007/978-3-030-66515-9

2021-01-01

Abstract:This edited volume illustrates the connections between machine learning techniques, black box optimization, and no-free lunch theorems. Each of the thirteen contributions focuses on the commonality and interdisciplinary concepts as well as the fundamentals needed to fully comprehend the impact of individual applications and problems. Current theoretical, algorithmic, and practical methods used are provided to stimulate a new effort towards innovative and efficient solutions. The book is intended for beginners who wish to achieve a broad overview of optimization methods and also for more experienced researchers as well as researchers in mathematics, optimization, operations research, quantitative logistics, data analysis, and statistics, who will benefit from access to a quick reference to key topics and methods. The coverage ranges from mathematically rigorous methods to heuristic and evolutionary approaches in an attempt to equip the reader with different viewpoints of the same problem.
Machine Learning Optimization Techniques: A Survey, Classification, Challenges, and Future Research Issues

Bian, Kewei,Priyadarshi, Rahul

DOI: https://doi.org/10.1007/s11831-024-10110-w

IF: 9.7

2024-03-30

Archives of Computational Methods in Engineering

Abstract:Optimization approaches in machine learning (ML) are essential for training models to obtain high performance across numerous domains. The article provides a comprehensive overview of ML optimization strategies, emphasizing their classification, obstacles, and potential areas for further study. We proceed with studying the historical progression of optimization methods, emphasizing significant developments and their influence on contemporary algorithms. We analyse the present research to identify widespread optimization algorithms and their uses in supervised learning, unsupervised learning, and reinforcement learning. Various common optimization constraints, including non-convexity, scalability issues, convergence problems, and concerns about robustness and generalization, are also explored. We suggest future research should focus on scalability problems, innovative optimization techniques, domain knowledge integration, and improving interpretability. The present study aims to provide an in-depth review of ML optimization by combining insights from historical advancements, literature evaluations, and current issues to guide future research efforts.

computer science, interdisciplinary applications,engineering, multidisciplinary,mathematics
Mathematics of Neural Networks (Lecture Notes Graduate Course)

Bart M. N. Smets

2024-03-06

Abstract:These are the lecture notes that accompanied the course of the same name that I taught at the Eindhoven University of Technology from 2021 to 2023. The course is intended as an introduction to neural networks for mathematics students at the graduate level and aims to make mathematics students interested in further researching neural networks. It consists of two parts: first a general introduction to deep learning that focuses on introducing the field in a formal mathematical way. The second part provides an introduction to the theory of Lie groups and homogeneous spaces and how it can be applied to design neural networks with desirable geometric equivariances. The lecture notes were made to be as self-contained as possible so as to accessible for any student with a moderate mathematics background. The course also included coding tutorials and assignments in the form of a set of Jupyter notebooks that are publicly available at https://gitlab.com/bsmetsjr/mathematics_of_neural_networks.

Artificial Intelligence,Machine Learning
Notes on Randomized Algorithms

James Aspnes

2024-05-08

Abstract:Lecture notes for the Yale Computer Science course CPSC 469/569 Randomized Algorithms. Suitable for use as a supplementary text for an introductory graduate or advanced undergraduate course on randomized algorithms. Discusses tools from probability theory, including random variables and expectations, union bound arguments, concentration bounds, applications of martingales and Markov chains, and the Lovász Local Lemma. Algorithmic topics include analysis of classic randomized algorithms such as Quicksort and Hoare's FIND, randomized tree data structures, hashing, Markov chain Monte Carlo sampling, randomized approximate counting, derandomization, quantum computing, and some examples of randomized distributed algorithms.

Data Structures and Algorithms
Tutorial on amortized optimization

Brandon Amos

DOI: https://doi.org/10.48550/arXiv.2202.00665

2023-04-24

Abstract:Optimization is a ubiquitous modeling tool and is often deployed in settings which repeatedly solve similar instances of the same problem. Amortized optimization methods use learning to predict the solutions to problems in these settings, exploiting the shared structure between similar problem instances. These methods have been crucial in variational inference and reinforcement learning and are capable of solving optimization problems many orders of magnitudes times faster than traditional optimization methods that do not use amortization. This tutorial presents an introduction to the amortized optimization foundations behind these advancements and overviews their applications in variational inference, sparse coding, gradient-based meta-learning, control, reinforcement learning, convex optimization, optimal transport, and deep equilibrium networks. The source code for this tutorial is available at <a class="link-external link-https" href="https://github.com/facebookresearch/amortized-optimization-tutorial" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence,Optimization and Control
Notes on Deep Learning Theory

Eugene A. Golikov

DOI: https://doi.org/10.48550/arXiv.2012.05760

2020-12-10

Abstract:These are the notes for the lectures that I was giving during Fall 2020 at the Moscow Institute of Physics and Technology (MIPT) and at the Yandex School of Data Analysis (YSDA). The notes cover some aspects of initialization, loss landscape, generalization, and a neural tangent kernel theory. While many other topics (e.g. expressivity, a mean-field theory, a double descent phenomenon) are missing in the current version, we plan to add them in future revisions.

Machine Learning,Artificial Intelligence
Introduction to Online Convex Optimization

Elad Hazan

2023-08-06

Abstract:This manuscript portrays optimization as a process. In many practical applications the environment is so complex that it is infeasible to lay out a comprehensive theoretical model and use classical algorithmic theory and mathematical optimization. It is necessary as well as beneficial to take a robust approach, by applying an optimization method that learns as one goes along, learning from experience as more aspects of the problem are observed. This view of optimization as a process has become prominent in varied fields and has led to some spectacular success in modeling and systems that are now part of our daily lives.

Machine Learning,Optimization and Control
Three lectures on free probability

Jonathan Novak,Michael LaCroix

DOI: https://doi.org/10.48550/arXiv.1205.2097

2012-05-10

Abstract:These are notes from a three-lecture mini-course on free probability given at MSRI in the Fall of 2010 and repeated a year later at Harvard. The lectures were aimed at mathematicians and mathematical physicists working in combinatorics, probability, and random matrix theory. The first lecture was a staged rediscovery of free independence from first principles, the second dealt with the additive calculus of free random variables, and the third focused on random matrix models.

Combinatorics,Mathematical Physics,Probability
Machine learning with neural networks

B. Mehlig

DOI: https://doi.org/10.1017/9781108860604

2021-10-27

Abstract:These are lecture notes for a course on machine learning with neural networks for scientists and engineers that I have given at Gothenburg University and Chalmers Technical University in Gothenburg, Sweden. The material is organised into three parts: Hopfield networks, supervised learning of labeled data, and learning algorithms for unlabeled data sets. Part I introduces stochastic recurrent networks: Hopfield networks and Boltzmann machines. The analysis of their learning rules sets the scene for the later parts. Part II describes supervised learning with multilayer perceptrons and convolutional neural networks. This part starts with a simple geometrical interpretation of the learning rule and leads to the recent successes of convolutional networks in object recognition, recurrent networks in language processing, and reservoir computers in time-series analysis. Part III explains what neural networks can learn about data that is not labeled. This part begins with a description of unsupervised learning techniques for clustering of data, non-linear projections, and embeddings. A section on autoencoders explains how to learn without labels using convolutional networks, and the last chapter is dedicated to reinforcement learning. The overall goal of the course is to explain the fundamental principles that allow neural networks to learn, emphasising ideas and concepts that are common to all three parts. The present version does not contain exercises (copyright owned by Cambridge University Press). The complete book is available at <a class="link-external link-https" href="https://www.cambridge.org/gb/academic/subjects/physics/statistical-physics/machine-learning-neural-networks-introduction-scientists-and-engineers?format=HB" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Statistical Mechanics
Foundations of Descriptive and Inferential Statistics

Henk van Elst

DOI: https://doi.org/10.13140/RG.2.1.2112.3044

2019-08-30

Abstract:These lecture notes were written with the aim to provide an accessible though technically solid introduction to the logic of systematical analyses of statistical data to both undergraduate and postgraduate students, in particular in the Social Sciences, Economics, and the Financial Services. They may also serve as a general reference for the application of quantitative--empirical research methods. In an attempt to encourage the adoption of an interdisciplinary perspective on quantitative problems arising in practice, the notes cover the four broad topics (i) descriptive statistical processing of raw data, (ii) elementary probability theory, (iii) the operationalisation of one-dimensional latent statistical variables according to Likert's widely used scaling approach, and (iv) null hypothesis significance testing within the frequentist approach to probability theory concerning (a) distributional differences of variables between subgroups of a target population, and (b) statistical associations between two variables. The relevance of effect sizes for making inferences is emphasised. These lecture notes are fully hyperlinked, thus providing a direct route to original scientific papers as well as to interesting biographical information. They also list many commands for running statistical functions and data analysis routines in the software packages R, SPSS, EXCEL and OpenOffice. The immediate involvement in actual data analysis practices is strongly recommended.

Applications,Other Statistics
Learning From An Optimization Viewpoint

Karthik Sridharan

DOI: https://doi.org/10.48550/arXiv.1204.4145

2012-04-19

Abstract:In this dissertation we study statistical and online learning problems from an optimization <a class="link-external link-http" href="http://viewpoint.The" rel="external noopener nofollow">this http URL</a> dissertation is divided into two parts : I. We first consider the question of learnability for statistical learning problems in the general learning setting. The question of learnability is well studied and fully characterized for binary classification and for real valued supervised learning problems using the theory of uniform convergence. However we show that for the general learning setting uniform convergence theory fails to characterize learnability. To fill this void we use stability of learning algorithms to fully characterize statistical learnability in the general setting. Next we consider the problem of online learning. Unlike the statistical learning framework there is a dearth of generic tools that can be used to establish learnability and rates for online learning problems in general. We provide online analogs to classical tools from statistical learning theory like Rademacher complexity, covering numbers, etc. We further use these tools to fully characterize learnability for online supervised learning problems. II. In the second part, for general classes of convex learning problems, we provide appropriate mirror descent (MD) updates for online and statistical learning of these problems. Further, we show that the the MD is near optimal for online convex learning and for most cases, is also near optimal for statistical convex learning. We next consider the problem of convex optimization and show that oracle complexity can be lower bounded by the so called fat-shattering dimension of the associated linear class. Thus we establish a strong connection between offline convex optimization problems and statistical learning problems. We also show that for a large class of high dimensional optimization problems, MD is in fact near optimal even for convex optimization.

Machine Learning,Computer Science and Game Theory
Bayesian Optimization for Machine Learning : A Practical Guidebook

Ian Dewancker,Michael McCourt,Scott Clark

DOI: https://doi.org/10.48550/arXiv.1612.04858

IF: 5.414

2016-12-14

Machine Learning

Abstract:The engineering of machine learning systems is still a nascent field; relying on a seemingly daunting collection of quickly evolving tools and best practices. It is our hope that this guidebook will serve as a useful resource for machine learning practitioners looking to take advantage of Bayesian optimization techniques. We outline four example machine learning problems that can be solved using open source machine learning libraries, and highlight the benefits of using Bayesian optimization in the context of these common machine learning applications.
An Introduction to Advanced Machine Learning : Meta Learning Algorithms, Applications and Promises

Farid Ghareh Mohammadi,M. Hadi Amini,Hamid R. Arabnia

DOI: https://doi.org/10.48550/arXiv.1908.09788

2019-08-27

Abstract:In [1, 2], we have explored the theoretical aspects of feature extraction optimization processes for solving largescale problems and overcoming machine learning limitations. Majority of optimization algorithms that have been introduced in [1, 2] guarantee the optimal performance of supervised learning, given offline and discrete data, to deal with curse of dimensionality (CoD) problem. These algorithms, however, are not tailored for solving emerging learning problems. One of the important issues caused by online data is lack of sufficient samples per class. Further, traditional machine learning algorithms cannot achieve accurate training based on limited distributed data, as data has proliferated and dispersed significantly. Machine learning employs a strict model or embedded engine to train and predict which still fails to learn unseen classes and sufficiently use online data. In this chapter, we introduce these challenges elaborately. We further investigate Meta-Learning (MTL) algorithm, and their application and promises to solve the emerging problems by answering how autonomous agents can learn to learn?.

Machine Learning

Lecture Notes: Optimization for Machine Learning

Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning

A Course in Dynamic Optimization

Lecture Notes on Free Probability

Introduction to Machine Learning: Class Notes 67577

Introduction to Machine Learning for the Sciences

Lecture Notes: Neural Network Architectures

Quantum algorithms for optimizers

Black Box Optimization, Machine Learning, and No-Free Lunch Theorems

Machine Learning Optimization Techniques: A Survey, Classification, Challenges, and Future Research Issues

Mathematics of Neural Networks (Lecture Notes Graduate Course)

Notes on Randomized Algorithms

Tutorial on amortized optimization

Notes on Deep Learning Theory

Introduction to Online Convex Optimization

Three lectures on free probability

Machine learning with neural networks

Foundations of Descriptive and Inferential Statistics

Learning From An Optimization Viewpoint

Bayesian Optimization for Machine Learning : A Practical Guidebook

An Introduction to Advanced Machine Learning : Meta Learning Algorithms, Applications and Promises