Machine Learning and Information Theory Concepts towards an AI Mathematician

Yoshua Bengio,Nikolay Malkin

DOI: https://doi.org/10.1090/bull/1839

2024-05-25

Bulletin of the American Mathematical Society

Abstract:Yoshua Bengio and Nikolay Malkin Bull. Amer. Math. Soc. 61 (), 457-469. , references and article information

mathematics

What problem does this paper attempt to address?

Information-Theoretic Foundations for Machine Learning

Hong Jun Jeon,Benjamin Van Roy

2024-08-20

Abstract:The staggering progress of machine learning in the past decade has been a sight to behold. In retrospect, it is both remarkable and unsettling that these milestones were achievable with little to no rigorous theory to guide experimentation. Despite this fact, practitioners have been able to guide their future experimentation via observations from previous large-scale empirical investigations. However, alluding to Plato's Allegory of the cave, it is likely that the observations which form the field's notion of reality are but shadows representing fragments of that reality. In this work, we propose a theoretical framework which attempts to answer what exists outside of the cave. To the theorist, we provide a framework which is mathematically rigorous and leaves open many interesting ideas for future exploration. To the practitioner, we provide a framework whose results are very intuitive, general, and which will help form principles to guide future investigations. Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. Our framework characterizes the performance of an optimal Bayesian learner, which considers the fundamental limits of information. Throughout this work, we derive very general theoretical results and apply them to derive insights specific to settings ranging from data which is independently and identically distributed under an unknown distribution, to data which is sequential, to data which exhibits hierarchical structure amenable to meta-learning. We conclude with a section dedicated to characterizing the performance of misspecified algorithms. These results are exciting and particularly relevant as we strive to overcome increasingly difficult machine learning challenges in this endlessly complex world.

Machine Learning,Artificial Intelligence
Machine Learning and Computational Mathematics

E. Weinan

DOI: https://doi.org/10.4208/cicp.OA-2020-0185

IF: 3.791

2020-01-01

Communications in Computational Physics

Abstract:Neural network-based machine learning is capable of approximating functions in very high dimension with unprecedented efficiency and accuracy. This has opened up many exciting new possibilities, not just in traditional areas of artificial intelligence, but also in scientific computing and computational science. At the same time, machine learning has also acquired the reputation of being a set of "black box" type of tricks, without fundamental principles. This has been a real obstacle for making further progress in machine learning. In this article, we try to address the following two very important questions: (1) How machine learning has already impacted and will further impact computational mathematics, scientific computing and computational science? (2) How computational mathematics, particularly numerical analysis, can impact machine learning? We describe some of the most important progress that has been made on these issues. Our hope is to put things into a perspective that will help to integrate machine learning with computational mathematics.
Information Theory and its Relation to Machine Learning

Bao-Gang Hu

DOI: https://doi.org/10.48550/arXiv.1501.04309

2015-01-18

Information Theory

Abstract:In this position paper, I first describe a new perspective on machine learning (ML) by four basic problems (or levels), namely, "What to learn?", "How to learn?", "What to evaluate?", and "What to adjust?". The paper stresses more on the first level of "What to learn?", or "Learning Target Selection". Towards this primary problem within the four levels, I briefly review the existing studies about the connection between information theoretical learning (ITL [1]) and machine learning. A theorem is given on the relation between the empirically-defined similarity measure and information measures. Finally, a conjecture is proposed for pursuing a unified mathematical interpretation to learning target selection.
Advancing mathematics by guiding human intuition with AI

Alex Davies,Petar Veličković,Lars Buesing,Sam Blackwell,Daniel Zheng,Nenad Tomašev,Richard Tanburn,Peter Battaglia,Charles Blundell,András Juhász,Marc Lackenby,Geordie Williamson,Demis Hassabis,Pushmeet Kohli

DOI: https://doi.org/10.1038/s41586-021-04086-x

IF: 64.8

2021-12-01

Nature

Abstract:Abstract The practice of mathematics involves discovering patterns and using these to formulate and prove conjectures, resulting in theorems. Since the 1960s, mathematicians have used computers to assist in the discovery of patterns and formulation of conjectures 1 , most famously in the Birch and Swinnerton-Dyer conjecture 2 , a Millennium Prize Problem 3 . Here we provide examples of new fundamental results in pure mathematics that have been discovered with the assistance of machine learning—demonstrating a method by which machine learning can aid mathematicians in discovering new conjectures and theorems. We propose a process of using machine learning to discover potential patterns and relations between mathematical objects, understanding them with attribution techniques and using these observations to guide intuition and propose conjectures. We outline this machine-learning-guided framework and demonstrate its successful application to current research questions in distinct areas of pure mathematics, in each case showing how it led to meaningful mathematical contributions on important open problems: a new connection between the algebraic and geometric structure of knots, and a candidate algorithm predicted by the combinatorial invariance conjecture for symmetric groups 4 . Our work may serve as a model for collaboration between the fields of mathematics and artificial intelligence (AI) that can achieve surprising results by leveraging the respective strengths of mathematicians and machine learning.

multidisciplinary sciences
The Mathematics of Artificial Intelligence

Gitta Kutyniok

DOI: https://doi.org/10.48550/arXiv.2203.08890

2022-03-17

Abstract:We currently witness the spectacular success of artificial intelligence in both science and public life. However, the development of a rigorous mathematical foundation is still at an early stage. In this survey article, which is based on an invited lecture at the International Congress of Mathematicians 2022, we will in particular focus on the current "workhorse" of artificial intelligence, namely deep neural networks. We will present the main theoretical directions along with several exemplary results and discuss key open problems.

Machine Learning,History and Overview
Introduction to Machine Learning

Laurent Younes

2024-09-04

Abstract:This book introduces the mathematical foundations and techniques that lead to the development and analysis of many of the algorithms that are used in machine learning. It starts with an introductory chapter that describes notation used throughout the book and serve at a reminder of basic concepts in calculus, linear algebra and probability and also introduces some measure theoretic terminology, which can be used as a reading guide for the sections that use these tools. The introductory chapters also provide background material on matrix analysis and optimization. The latter chapter provides theoretical support to many algorithms that are used in the book, including stochastic gradient descent, proximal methods, etc. After discussing basic concepts for statistical prediction, the book includes an introduction to reproducing kernel theory and Hilbert space techniques, which are used in many places, before addressing the description of various algorithms for supervised statistical learning, including linear methods, support vector machines, decision trees, boosting, or neural networks. The subject then switches to generative methods, starting with a chapter that presents sampling methods and an introduction to the theory of Markov chains. The following chapter describe the theory of graphical models, an introduction to variational methods for models with latent variables, and to deep-learning based generative models. The next chapters focus on unsupervised learning methods, for clustering, factor analysis and manifold learning. The final chapter of the book is theory-oriented and discusses concentration inequalities and generalization bounds.

Machine Learning
Mathematical Perspective of Machine Learning

Yarema Boryshchak

DOI: https://doi.org/10.48550/arXiv.2007.01503

IF: 5.414

2020-07-03

Machine Learning

Abstract:We take a closer look at some theoretical challenges of Machine Learning as a function approximation, gradient descent as the default optimization algorithm, limitations of fixed length and width networks and a different approach to RNNs from a mathematical perspective.
Computational principles of intelligence: learning and reasoning with neural networks

Abel Torres Montoya

DOI: https://doi.org/10.48550/arXiv.2012.09477

2020-12-17

Abstract:Despite significant achievements and current interest in machine learning and artificial intelligence, the quest for a theory of intelligence, allowing general and efficient problem solving, has done little progress. This work tries to contribute in this direction by proposing a novel framework of intelligence based on three principles. First, the generative and mirroring nature of learned representations of inputs. Second, a grounded, intrinsically motivated and iterative process for learning, problem solving and imagination. Third, an ad hoc tuning of the reasoning mechanism over causal compositional representations using inhibition rules. Together, those principles create a systems approach offering interpretability, continuous learning, common sense and more. This framework is being developed from the following perspectives: as a general problem solving method, as a human oriented tool and finally, as model of information processing in the brain.

Artificial Intelligence,Machine Learning
Quantum Mathematics in Artificial Intelligence

Dominic Widdows,Kirsty Kitto,Trevor Cohen

DOI: https://doi.org/10.1613/jair.1.12702

2021-12-17

Abstract:In the decade since 2010, successes in artificial intelligence have been at the forefront of computer science and technology, and vector space models have solidified a position at the forefront of artificial intelligence. At the same time, quantum computers have become much more powerful, and announcements of major advances are frequently in the news. The mathematical techniques underlying both these areas have more in common than is sometimes realized. Vector spaces took a position at the axiomatic heart of quantum mechanics in the 1930s, and this adoption was a key motivation for the derivation of logic and probability from the linear geometry of vector spaces. Quantum interactions between particles are modelled using the tensor product, which is also used to express objects and operations in artificial neural networks. This paper describes some of these common mathematical areas, including examples of how they are used in artificial intelligence (AI), particularly in automated reasoning and natural language processing (NLP). Techniques discussed include vector spaces, scalar products, subspaces and implication, orthogonal projection and negation, dual vectors, density matrices, positive operators, and tensor products. Application areas include information retrieval, categorization and implication, modelling word-senses and disambiguation, inference in knowledge bases, and semantic composition. Some of these approaches can potentially be implemented on quantum hardware. Many of the practical steps in this implementation are in early stages, and some are already realized. Explaining some of the common mathematical tools can help researchers in both AI and quantum computing further exploit these overlaps, recognizing and exploring new directions along the way.

Artificial Intelligence,Computation and Language,Information Retrieval
Importance of the Mathematical Foundations of Machine Learning Methods for Scientific and Engineering Applications

Paul J. Atzberger

DOI: https://doi.org/10.48550/arXiv.1808.02213

IF: 5.414

2018-08-07

Machine Learning

Abstract:There has been a lot of recent interest in adopting machine learning methods for scientific and engineering applications. This has in large part been inspired by recent successes and advances in the domains of Natural Language Processing (NLP) and Image Classification (IC). However, scientific and engineering problems have their own unique characteristics and requirements raising new challenges for effective design and deployment of machine learning approaches. There is a strong need for further mathematical developments on the foundations of machine learning methods to increase the level of rigor of employed methods and to ensure more reliable and interpretable results. Also as reported in the recent literature on state-of-the-art results and indicated by the No Free Lunch Theorems of statistical learning theory incorporating some form of inductive bias and domain knowledge is essential to success. Consequently, even for existing and widely used methods there is a strong need for further mathematical work to facilitate ways to incorporate prior scientific knowledge and related inductive biases into learning frameworks and algorithms. We briefly discuss these topics and discuss some ideas proceeding in this direction.
Introduction to Machine Learning: Class Notes 67577

Amnon Shashua

DOI: https://doi.org/10.48550/arXiv.0904.3664

IF: 5.414

2009-04-23

Machine Learning

Abstract:Introduction to Machine learning covering Statistical Inference (Bayes, EM, ML/MaxEnt duality), algebraic and spectral methods (PCA, LDA, CCA, Clustering), and PAC learning (the Formal model, VC dimension, Double Sampling theorem).
A New Information Theory of Certainty for Machine Learning

Arthur Jun Zhang

2023-04-25

Abstract:Claude Shannon coined entropy to quantify the uncertainty of a random distribution for communication coding theory. We observe that the uncertainty nature of entropy also limits its direct usage in mathematical modeling. Therefore we propose a new concept troenpy,as the canonical dual of entropy, to quantify the certainty of the underlying distribution. We demonstrate two applications in machine learning. The first is for the classical document classification, we develop a troenpy based weighting scheme to leverage the document class label. The second is a self-troenpy weighting scheme for sequential data and show that it can be easily included in neural network based language models and achieve dramatic perplexity reduction. We also define quantum troenpy as the dual of the Von Neumann entropy to quantify the certainty of quantum systems.

Information Theory,Artificial Intelligence,Computation and Language,Information Retrieval,Machine Learning
Category Theory in Machine Learning

Dan Shiebler,Bruno Gavranović,Paul Wilson

DOI: https://doi.org/10.48550/arXiv.2106.07032

IF: 5.414

2021-06-13

Machine Learning

Abstract:Over the past two decades machine learning has permeated almost every realm of technology. At the same time, many researchers have begun using category theory as a unifying language, facilitating communication between different scientific disciplines. It is therefore unsurprising that there is a burgeoning interest in applying category theory to machine learning. We aim to document the motivations, goals and common themes across these applications. We touch on gradient-based learning, probability, and equivariant learning.
Entropy, concentration, and learning: a statistical mechanics primer

Akshay Balsubramani

2024-09-27

Abstract:Artificial intelligence models trained through loss minimization have demonstrated significant success, grounded in principles from fields like information theory and statistical physics. This work explores these established connections through the lens of statistical mechanics, starting from first-principles sample concentration behaviors that underpin AI and machine learning. Our development of statistical mechanics for modeling highlights the key role of exponential families, and quantities of statistics, physics, and information theory.

Machine Learning,Statistical Mechanics,Artificial Intelligence,Information Theory
Learning from Learning Machines: Optimisation, Rules, and Social Norms

Travis LaCroix,Yoshua Bengio

DOI: https://doi.org/10.48550/arXiv.2001.00006

2019-12-29

Computers and Society

Abstract:There is an analogy between machine learning systems and economic entities in that they are both adaptive, and their behaviour is specified in a more-or-less explicit way. It appears that the area of AI that is most analogous to the behaviour of economic entities is that of morally good decision-making, but it is an open question as to how precisely moral behaviour can be achieved in an AI system. This paper explores the analogy between these two complex systems, and we suggest that a clearer understanding of this apparent analogy may help us forward in both the socio-economic domain and the AI domain: known results in economics may help inform feasible solutions in AI safety, but also known results in AI may inform economic policy. If this claim is correct, then the recent successes of deep learning for AI suggest that more implicit specifications work better than explicit ones for solving such problems.
Artificial Intelligence in Mathematical Modeling of Complex Systems

Ting Zhao

DOI: https://doi.org/10.4108/eetel.5256

2024-03-28

EAI Endorsed Transactions on e-Learning

Abstract:This article introduces artificial intelligence techniques in mathematical modelling of complex systems and their applications. Mathematical modelling of complex systems is a method of studying the structure and behaviour of complex systems, aiming to understand interactions and nonlinear effects in the system. Commonly used modelling methods include system dynamics, network theory, and algebraic methods. Artificial intelligence technologies include machine learning and deep learning, which can be used for tasks such as prediction and classification, anomaly detection, optimization and decision-making. In mathematical modelling of complex systems, artificial intelligence technology can learn system patterns and laws from large amounts of data, and can be applied to image and speech recognition, time series analysis and other fields. Deep learning and machine learning are important branches of artificial intelligence. They realize the modelling and analysis of complex systems by building neural network models. Data-driven modelling is a modelling method based on actual data that, combined with traditional theoretical modelling, can better describe and predict the behaviour of complex systems. Self-control of complex systems means that the system realizes its own optimization and adjustment through adaptive control algorithms and feedback mechanisms. In summary, artificial intelligence technology has broad application prospects in mathematical modelling of complex systems and will provide new tools and methods for in-depth understanding and solving problems in complex systems.
Machine Learning in Artificial Intelligence: Towards a Common Understanding

Niklas Kühl,Marc Goutier,Robin Hirt,Gerhard Satzger

DOI: https://doi.org/10.48550/arXiv.2004.04686

2020-03-28

Abstract:The application of "machine learning" and "artificial intelligence" has become popular within the last decade. Both terms are frequently used in science and media, sometimes interchangeably, sometimes with different meanings. In this work, we aim to clarify the relationship between these terms and, in particular, to specify the contribution of machine learning to artificial intelligence. We review relevant literature and present a conceptual framework which clarifies the role of machine learning to build (artificial) intelligent agents. Hence, we seek to provide more terminological clarity and a starting point for (interdisciplinary) discussions and future research.

Machine Learning,Artificial Intelligence
Towards a Mathematical Understanding of Neural Network-Based Machine Learning: What We Know and What We Don't

E. Weinan,Chao Ma,Lei Wu,Stephan Wojtowytsch

DOI: https://doi.org/10.4208/csiam-am.so-2020-0002

2020-01-01

CSIAM Transactions on Applied Mathematics

Abstract:The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning. In the tradition of good old applied mathematics, we will not only give attention to rigorous mathematical results, but also the insight we have gained from careful numerical experiments as well as the analysis of simplified models. Along the way, we also list the open problems which we believe to be the most important topics for further study. This is not a complete overview over this quickly moving field, but we hope to provide a perspective which may be helpful especially to new researchers in the area.
The Challenges of Machine Learning: A Critical Review

Enrico Barbierato,Alice Gatti

DOI: https://doi.org/10.3390/electronics13020416

IF: 2.9

2024-01-20

Electronics

Abstract:The concept of learning has multiple interpretations, ranging from acquiring knowledge or skills to constructing meaning and social development. Machine Learning (ML) is considered a branch of Artificial Intelligence (AI) and develops algorithms that can learn from data and generalize their judgment to new observations by exploiting primarily statistical methods. The new millennium has seen the proliferation of Artificial Neural Networks (ANNs), a formalism able to reach extraordinary achievements in complex problems such as computer vision and natural language recognition. In particular, designers claim that this formalism has a strong resemblance to the way the biological neurons operate. This work argues that although ML has a mathematical/statistical foundation, it cannot be strictly regarded as a science, at least from a methodological perspective. The main reason is that ML algorithms have notable prediction power although they cannot necessarily provide a causal explanation about the achieved predictions. For example, an ANN could be trained on a large dataset of consumer financial information to predict creditworthiness. The model takes into account various factors like income, credit history, debt, spending patterns, and more. It then outputs a credit score or a decision on credit approval. However, the complex and multi-layered nature of the neural network makes it almost impossible to understand which specific factors or combinations of factors the model is using to arrive at its decision. This lack of transparency can be problematic, especially if the model denies credit and the applicant wants to know the specific reasons for the denial. The model's "black box" nature means it cannot provide a clear explanation or breakdown of how it weighed the various factors in its decision-making process. Secondly, this work rejects the belief that a machine can simply learn from data, either in supervised or unsupervised mode, just by applying statistical methods. The process of learning is much more complex, as it requires the full comprehension of a learned ability or skill. In this sense, further ML advancements, such as reinforcement learning and imitation learning denote encouraging similarities to similar cognitive skills used in human learning.

engineering, electrical & electronic,computer science, information systems,physics, applied
What Machine Learning Tells Us About the Mathematical Structure of Concepts

Jun Otsuka

2024-08-28

Abstract:This paper examines the connections among various approaches to understanding concepts in philosophy, cognitive science, and machine learning, with a particular focus on their mathematical nature. By categorizing these approaches into Abstractionism, the Similarity Approach, the Functional Approach, and the Invariance Approach, the study highlights how each framework provides a distinct mathematical perspective for modeling concepts. The synthesis of these approaches bridges philosophical theories and contemporary machine learning models, providing a comprehensive framework for future research. This work emphasizes the importance of interdisciplinary dialogue, aiming to enrich our understanding of the complex relationship between human cognition and artificial intelligence.

Artificial Intelligence

Machine Learning and Information Theory Concepts towards an AI Mathematician

Information-Theoretic Foundations for Machine Learning

Machine Learning and Computational Mathematics

Information Theory and its Relation to Machine Learning

Advancing mathematics by guiding human intuition with AI

The Mathematics of Artificial Intelligence

Introduction to Machine Learning

Mathematical Perspective of Machine Learning

Computational principles of intelligence: learning and reasoning with neural networks

Quantum Mathematics in Artificial Intelligence

Importance of the Mathematical Foundations of Machine Learning Methods for Scientific and Engineering Applications

Introduction to Machine Learning: Class Notes 67577

A New Information Theory of Certainty for Machine Learning

Category Theory in Machine Learning

Entropy, concentration, and learning: a statistical mechanics primer

Learning from Learning Machines: Optimisation, Rules, and Social Norms

Artificial Intelligence in Mathematical Modeling of Complex Systems

Machine Learning in Artificial Intelligence: Towards a Common Understanding

Towards a Mathematical Understanding of Neural Network-Based Machine Learning: What We Know and What We Don't

The Challenges of Machine Learning: A Critical Review

What Machine Learning Tells Us About the Mathematical Structure of Concepts