Mathematical Perspective of Machine Learning

Yarema Boryshchak

DOI: https://doi.org/10.48550/arXiv.2007.01503

IF: 5.414

2020-07-03

Machine Learning

Abstract:We take a closer look at some theoretical challenges of Machine Learning as a function approximation, gradient descent as the default optimization algorithm, limitations of fixed length and width networks and a different approach to RNNs from a mathematical perspective.

What problem does this paper attempt to address?

A mathematical perspective of machine learning

E Weinan

2022-01-01

Abstract:What lies at the heart of modern neural network-based machine learning is the ability to approximate very high dimensional functions with good accuracy. This opens up two major avenues of research. The first is to develop machine learning-based algorithms for scientific problems that suffer from the curse of dimensionality. The second is to build a theoretical framework that helps us to form a better foundation for machine learning. For the latter, the most important questions that need to be addressed include: Why do neural network models work so well in high dimension? Why does their performance depend so sensitively on the choice of the hyperparameters? Can we develop more robust and equally accurate new machine learning models and algorithms? In this article, we review some of the major progresses made in these directions.
Machine learning and computational mathematics

E Weinan

2020-09-01

Abstract:Neural network-based machine learning is capable of approximating functions in very high dimension with unprecedented efficiency and accuracy. This has opened up many exciting new possibilities, not just in traditional areas of artificial intelligence, but also in scientific computing and computational science. At the same time, machine learning has also acquired the reputation of being a set of "black box" type of tricks, without fundamental principles. This has been a real obstacle for making further progress in machine learning. In this article, we try to address the following two very important questions: (1) How machine learning has already impacted and will further impact computational mathematics, scientific computing and computational science? (2) How computational mathematics, particularly numerical analysis, {can} impact machine learning? We describe some of the most important progress that has been made on these issues. Our hope is to put things into a perspective that will help to integrate machine learning with computational mathematics.
Towards a Mathematical Understanding of Neural Network-Based Machine Learning: What We Know and What We Don't

E. Weinan,Chao Ma,Lei Wu,Stephan Wojtowytsch

DOI: https://doi.org/10.4208/csiam-am.so-2020-0002

2020-01-01

CSIAM Transactions on Applied Mathematics

Abstract:The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning. In the tradition of good old applied mathematics, we will not only give attention to rigorous mathematical results, but also the insight we have gained from careful numerical experiments as well as the analysis of simplified models. Along the way, we also list the open problems which we believe to be the most important topics for further study. This is not a complete overview over this quickly moving field, but we hope to provide a perspective which may be helpful especially to new researchers in the area.
Mathematical analysis of machine learning algorithms

Tong Zhang

2023-08-10

Abstract:The mathematical theory of machine learning not only explains the current algorithms but can also motivate principled approaches for the future. This self-contained textbook introduces students and researchers of AI to the main mathematical techniques used to analyze machine learning algorithms, with motivations and applications. Topics covered include the analysis of supervised learning algorithms in the iid setting, the analysis of neural networks (eg neural tangent kernel and mean-field analysis), and the analysis of machine learning algorithms in the sequential decision setting (eg online learning, bandit problems, and reinforcement learning). Students will learn the basic mathematical tools used in the theoretical analysis of these machine learning problems and how to apply them to the analysis of various concrete algorithms. This textbook is perfect for readers who have some background knowledge of basic machine learning methods, but want to gain sufficient technical knowledge to understand research papers in theoretical machine learning.
The Modern Mathematics of Deep Learning

Julius Berner,Philipp Grohs,Gitta Kutyniok,Philipp Petersen

DOI: https://doi.org/10.48550/arXiv.2105.04026

IF: 5.414

2021-05-09

Machine Learning

Abstract:We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.
Dynamical Perspective of Machine Learning: an Introductory Review

Yiming Zhang,Haoran Li,Chenlong Deng,Fuyong Wang,Yuan-Hua Ni

DOI: https://doi.org/10.1109/icca51439.2020.9264352

2020-01-01

Abstract:In this paper, we will briefly introduce the recent progress on machine learning based on the dynamical perspective. By combining machine learning with the dynamics, we get many continuous time algorithms. By discretizing the continuous time dynamic equations, we can not only review the existing algorithms, but also get new discrete-time algorithms. Firstly, gradient based algorithms is presented, which include the accelerated algorithm, policy gradient algorithm and stochastic gradient algorithm. Then, we come to the non-gradient algorithms, including the Pontryagins maximum principle based successive approximation algorithm and the optimal tensor algorithm. Next, continuous-time perspective of machine learning is reviewed, which to some extent establish a theoretical basis for machine learning. And in the end, we propose the advantage and the disadvantage of the above achievements, and try to give our own opinion about the application prospect and the development direction.
Introduction to Machine Learning

Laurent Younes

2024-09-04

Abstract:This book introduces the mathematical foundations and techniques that lead to the development and analysis of many of the algorithms that are used in machine learning. It starts with an introductory chapter that describes notation used throughout the book and serve at a reminder of basic concepts in calculus, linear algebra and probability and also introduces some measure theoretic terminology, which can be used as a reading guide for the sections that use these tools. The introductory chapters also provide background material on matrix analysis and optimization. The latter chapter provides theoretical support to many algorithms that are used in the book, including stochastic gradient descent, proximal methods, etc. After discussing basic concepts for statistical prediction, the book includes an introduction to reproducing kernel theory and Hilbert space techniques, which are used in many places, before addressing the description of various algorithms for supervised statistical learning, including linear methods, support vector machines, decision trees, boosting, or neural networks. The subject then switches to generative methods, starting with a chapter that presents sampling methods and an introduction to the theory of Markov chains. The following chapter describe the theory of graphical models, an introduction to variational methods for models with latent variables, and to deep-learning based generative models. The next chapters focus on unsupervised learning methods, for clustering, factor analysis and manifold learning. The final chapter of the book is theory-oriented and discusses concentration inequalities and generalization bounds.

Machine Learning
Importance of the Mathematical Foundations of Machine Learning Methods for Scientific and Engineering Applications

Paul J. Atzberger

DOI: https://doi.org/10.48550/arXiv.1808.02213

IF: 5.414

2018-08-07

Machine Learning

Abstract:There has been a lot of recent interest in adopting machine learning methods for scientific and engineering applications. This has in large part been inspired by recent successes and advances in the domains of Natural Language Processing (NLP) and Image Classification (IC). However, scientific and engineering problems have their own unique characteristics and requirements raising new challenges for effective design and deployment of machine learning approaches. There is a strong need for further mathematical developments on the foundations of machine learning methods to increase the level of rigor of employed methods and to ensure more reliable and interpretable results. Also as reported in the recent literature on state-of-the-art results and indicated by the No Free Lunch Theorems of statistical learning theory incorporating some form of inductive bias and domain knowledge is essential to success. Consequently, even for existing and widely used methods there is a strong need for further mathematical work to facilitate ways to incorporate prior scientific knowledge and related inductive biases into learning frameworks and algorithms. We briefly discuss these topics and discuss some ideas proceeding in this direction.
Regression Models for Machine Learning

Pengfei Wei,Michael Beer

2023-01-01

Abstract:This chapter investigates the regression models and methods for machine learning in engineering computations, from both non-Bayesian and Bayesian perspectives. The non-Bayesian regression models, including the least square regression, ridge regression, and support vector regression, equipped or not equipped with kernel trick, are first examined as they share the same principle, which is to find an element in the parametrically indexed hypothesis set by minimizing specific empirical loss functions. The Bayesian regression models are then investigated from the views of parametric space, and then functional space, all of which aim at assigning probabilistic beliefs on the elements of the hypothesis set using Bayesian inference. It is highlighted that, with specific assumptions, the Bayesian models provide measures of the prediction errors for unobserved points. This feature allows to develop a unique learning skill, i.e. active learning, which aims at devising optimal design strategies for minimizing the number of simulator calls, especially when each call is computationally cumbersome. This is shown to be effective when applied to cutting-edge research on Bayesian numerical analysis such as Bayesian cubature and Bayesian reliability assessment examined in this chapter. Both theoretical details and numerical examples are presented for examining each method.
Perspective on machine learning for advancing fluid mechanics

M. P. Brenner,J. D. Eldredge,J. B. Freund

DOI: https://doi.org/10.1103/physrevfluids.4.100501

IF: 2.895

2019-10-16

Physical Review Fluids

Abstract:A perspective is presented on how machine learning (ML), with its burgeoning popularity and the increasing availability of portable implementations, might advance fluid mechanics. As with any numerical or experimental method, ML methods have strengths and limitations, which are acknowledged. Their potential impact is high so long as outcomes are held to the long-standing critical standards that should guide studies of flow physics.

physics, fluids & plasmas
Mathematical Background of Machine Learning

Volker Liermann,Sangmeng Li,Victoria Dobryashkina

DOI: https://doi.org/10.1007/978-3-030-23719-6_16

2019-01-01

Abstract:This chapter classifies the different Machine learning algorithms into domains and provides a formal definition of machine learning. In addition, the chapter describes briefly a common set of the classic machine learning techniques. These sets span from time series forecasting to different clustering methods including trees and Bayesian networks. The special domain of deep learning is addressed in the following chapter (Liermann, Li, & Schaudinnus, Deep learning—An introduction, 2019b).
What Machine Learning Tells Us About the Mathematical Structure of Concepts

Jun Otsuka

2024-08-28

Abstract:This paper examines the connections among various approaches to understanding concepts in philosophy, cognitive science, and machine learning, with a particular focus on their mathematical nature. By categorizing these approaches into Abstractionism, the Similarity Approach, the Functional Approach, and the Invariance Approach, the study highlights how each framework provides a distinct mathematical perspective for modeling concepts. The synthesis of these approaches bridges philosophical theories and contemporary machine learning models, providing a comprehensive framework for future research. This work emphasizes the importance of interdisciplinary dialogue, aiming to enrich our understanding of the complex relationship between human cognition and artificial intelligence.

Artificial Intelligence
The Challenges of Machine Learning: A Critical Review

Enrico Barbierato,Alice Gatti

DOI: https://doi.org/10.3390/electronics13020416

IF: 2.9

2024-01-20

Electronics

Abstract:The concept of learning has multiple interpretations, ranging from acquiring knowledge or skills to constructing meaning and social development. Machine Learning (ML) is considered a branch of Artificial Intelligence (AI) and develops algorithms that can learn from data and generalize their judgment to new observations by exploiting primarily statistical methods. The new millennium has seen the proliferation of Artificial Neural Networks (ANNs), a formalism able to reach extraordinary achievements in complex problems such as computer vision and natural language recognition. In particular, designers claim that this formalism has a strong resemblance to the way the biological neurons operate. This work argues that although ML has a mathematical/statistical foundation, it cannot be strictly regarded as a science, at least from a methodological perspective. The main reason is that ML algorithms have notable prediction power although they cannot necessarily provide a causal explanation about the achieved predictions. For example, an ANN could be trained on a large dataset of consumer financial information to predict creditworthiness. The model takes into account various factors like income, credit history, debt, spending patterns, and more. It then outputs a credit score or a decision on credit approval. However, the complex and multi-layered nature of the neural network makes it almost impossible to understand which specific factors or combinations of factors the model is using to arrive at its decision. This lack of transparency can be problematic, especially if the model denies credit and the applicant wants to know the specific reasons for the denial. The model's "black box" nature means it cannot provide a clear explanation or breakdown of how it weighed the various factors in its decision-making process. Secondly, this work rejects the belief that a machine can simply learn from data, either in supervised or unsupervised mode, just by applying statistical methods. The process of learning is much more complex, as it requires the full comprehension of a learned ability or skill. In this sense, further ML advancements, such as reinforcement learning and imitation learning denote encouraging similarities to similar cognitive skills used in human learning.

engineering, electrical & electronic,computer science, information systems,physics, applied
A gentle introduction to gradient-based optimization and variational inequalities for machine learning

Neha S Wadia,Yatin Dandi,Michael I Jordan

DOI: https://doi.org/10.1088/1742-5468/ad3194

2024-10-31

Journal of Statistical Mechanics Theory and Experiment

Abstract:The rapid progress in machine learning in recent years has been based on a highly productive connection to gradient-based optimization. Further progress hinges in part on a shift in focus from pattern recognition to decision-making and multi-agent problems. In these broader settings, new mathematical challenges emerge that involve equilibria and game theory instead of optima. Gradient-based methods remain essential—given the high dimensionality and large scale of machine-learning problems—but simple gradient descent is no longer the point of departure for algorithm design. We provide a gentle introduction to a broader framework for gradient-based algorithms in machine learning, beginning with saddle points and monotone games, and proceeding to general variational inequalities. While we provide convergence proofs for several of the algorithms that we present, our main focus is that of providing motivation and intuition.

physics, mathematical,mechanics
Perspective: Machine learning in experimental solid mechanics

N.R. Brodnik,C. Muir,N. Tulshibagwale,J. Rossin,M.P. Echlin,C.M. Hamel,S.L.B. Kramer,T.M. Pollock,J.D. Kiser,C. Smith,S.H. Daly

DOI: https://doi.org/10.1016/j.jmps.2023.105231

IF: 5.582

2023-02-01

Journal of the Mechanics and Physics of Solids

Abstract:Experimental solid mechanics is at a pivotal point where machine learning (ML) approaches are rapidly proliferating into the discovery process, due to significant advances in data storage and processing capabilities. Much of the ML that is being adopted by the mechanics community was initially developed for application outside of science and engineering, and has the potential to produce results of questionable physical validity. To ensure that these data-driven approaches are trustworthy, there is a clear need to embed physical principles into their architectures, to evaluate and compare ML frameworks against benchmark datasets, and to test their broader extensibility. Frameworks must be grounded in a clear objective, quantifiable error, and a well-defined scope of extensibility. These principles enable ML models with a wide range of architectures to be meaningfully categorized, compared, evaluated, and extended to broader experimental and computational frameworks. Application of these principles are demonstrated through an investigation of ML models in two different use cases, acoustic emission and resonant ultrasound spectroscopy, along with a discussion of outlooks for the future of trustworthy ML in experimental mechanics.

mechanics,materials science, multidisciplinary,physics, condensed matter
Machine learning in physics and geometry

Yang-Hui He,Elli Heyes,Edward Hirst

DOI: https://doi.org/10.1016/bs.host.2023.06.002

2023-08-09

Handbook of Statistics

Abstract:We survey some recent applications of machine learning to problems in geometry and theoretical physics. Pure mathematical data has been compiled over the last few decades by the community, and experiments in supervised, semi-supervised, and unsupervised machine learning have found surprising success. We thus advocate the program of machine learning mathematical structures and formulating conjectures via pattern recognition, in other words using artificial intelligence to help one do mathematics.

English Else
Towards the Automatic Mathematician

Markus N. Rabe,Christian Szegedy

DOI: https://doi.org/10.1007/978-3-030-79876-5_2

2021-01-01

Abstract:Abstract Over the recent years deep learning has found successful applications in mathematical reasoning. Today, we can predict fine-grained proof steps, relevant premises, and even useful conjectures using neural networks. This extended abstract summarizes recent developments of machine learning in mathematical reasoning and the vision of the N2Formal group at Google Research to create an automatic mathematician. The second part discusses the key challenges on the road ahead.
From the String Landscape to the Mathematical Landscape: a Machine-Learning Outlook

Yang-Hui He

DOI: https://doi.org/10.48550/arXiv.2202.06086

2022-02-12

Abstract:We review the recent programme of using machine-learning to explore the landscape of mathematical problems. With this paradigm as a model for human intuition - complementary to and in contrast with the more formalistic approach of automated theorem proving - we highlight some experiments on how AI helps with conjecture formulation, pattern recognition and computation.

High Energy Physics - Theory,Machine Learning,History and Overview

Mathematical Perspective of Machine Learning

A mathematical perspective of machine learning

Machine learning and computational mathematics

Towards a Mathematical Understanding of Neural Network-Based Machine Learning: What We Know and What We Don't

Mathematical analysis of machine learning algorithms

The Modern Mathematics of Deep Learning

Dynamical Perspective of Machine Learning: an Introductory Review

Introduction to Machine Learning

Importance of the Mathematical Foundations of Machine Learning Methods for Scientific and Engineering Applications

Regression Models for Machine Learning

Perspective on machine learning for advancing fluid mechanics

Mathematical Background of Machine Learning

What Machine Learning Tells Us About the Mathematical Structure of Concepts

The Challenges of Machine Learning: A Critical Review

A gentle introduction to gradient-based optimization and variational inequalities for machine learning

Perspective: Machine learning in experimental solid mechanics

Machine learning in physics and geometry

Towards the Automatic Mathematician

From the String Landscape to the Mathematical Landscape: a Machine-Learning Outlook