Deep Learning: An Introduction for Applied Mathematicians

Catherine F. Higham,Desmond J. Higham

DOI: https://doi.org/10.1137/18M1165748

IF: 10.2

2019-11-06

SIAM Review

Abstract:Multilayered artificial neural networks are becoming a pervasive tool in a host of application fields. At the heart of this deep learning revolution are familiar concepts from applied and computati...

mathematics, applied

What problem does this paper attempt to address?

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

Arnulf Jentzen,Benno Kuckuck,Philippe von Wurstemberger

2023-10-31

Abstract:This book aims to provide an introduction to the topic of deep learning algorithms. We review essential components of deep learning algorithms in full mathematical detail including different artificial neural network (ANN) architectures (such as fully-connected feedforward ANNs, convolutional ANNs, recurrent ANNs, residual ANNs, and ANNs with batch normalization) and different optimization algorithms (such as the basic stochastic gradient descent (SGD) method, accelerated methods, and adaptive methods). We also cover several theoretical aspects of deep learning algorithms such as approximation capacities of ANNs (including a calculus for ANNs), optimization theory (including Kurdyka-Łojasiewicz inequalities), and generalization errors. In the last part of the book some deep learning approximation methods for PDEs are reviewed including physics-informed neural networks (PINNs) and deep Galerkin methods. We hope that this book will be useful for students and scientists who do not yet have any background in deep learning at all and would like to gain a solid foundation as well as for practitioners who would like to obtain a firmer mathematical understanding of the objects and methods considered in deep learning.

Machine Learning,Artificial Intelligence,Numerical Analysis,Probability
Deep Learning: A Tutorial

Nick Polson,Vadim Sokolov

2023-10-10

Abstract:Our goal is to provide a review of deep learning methods which provide insight into structured high-dimensional data. Rather than using shallow additive architectures common to most statistical models, deep learning uses layers of semi-affine input transformations to provide a predictive rule. Applying these layers of transformations leads to a set of attributes (or, features) to which probabilistic statistical methods can be applied. Thus, the best of both worlds can be achieved: scalable prediction rules fortified with uncertainty quantification, where sparse regularization finds the features.

Machine Learning
Deep Learning

Yann LeCun,Geoffrey E. Hinton,Yoshua Bengio

Abstract:Machine-learning technology powers many aspects of modern society: from web searches to content filtering on social networks to recommendations on e-commerce websites, and it is increasingly present in consumer products such as cameras and smartphones. Machine-learning systems are used to identify objects in images, transcribe speech into text, match news items, posts or products with users’ interests, and select relevant results of search. Increasingly, these applications make use of a class of techniques called deep learning. Conventional machine-learning techniques were limited in their ability to process natural data in their raw form. For decades, constructing a pattern-recognition or machine-learning system required careful engineering and considerable domain expertise to design a feature extractor that transformed the raw data (such as the pixel values of an image) into a suitable internal representation or feature vector from which the learning subsystem, often a classifier, could detect or classify patterns in the input. Representation learning is a set of methods that allows a machine to be fed with raw data and to automatically discover the representations needed for detection or classification. Deep-learning methods are representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules that each transform the representation at one level (starting with the raw input) into a representation at a higher, slightly more abstract level. With the composition of enough such transformations, very complex functions can be learned. For classification tasks, higher layers of representation amplify aspects of the input that are important for discrimination and suppress irrelevant variations. An image, for example, comes in the form of an array of pixel values, and the learned features in the first layer of representation typically represent the presence or absence of edges at particular orientations and locations in the image. The second layer typically detects motifs by spotting particular arrangements of edges, regardless of small variations in the edge positions. The third layer may assemble motifs into larger combinations that correspond to parts of familiar objects, and subsequent layers would detect objects as combinations of these parts. The key aspect of deep learning is that these layers of features are not designed by human engineers: they are learned from data using a general-purpose learning procedure. Deep learning is making major advances in solving problems that have resisted the best attempts of the artificial intelligence community for many years. It has turned out to be very good at discovering intricate structures in high-dimensional data and is therefore applicable to many domains of science, business and government. In addition to beating records in image recognition and speech recognition, it has beaten other machine-learning techniques at predicting the activity of potential drug molecules, analysing particle accelerator data, reconstructing brain circuits, and predicting the effects of mutations in non-coding DNA on gene expression and disease. Perhaps more surprisingly, deep learning has produced extremely promising results for various tasks in natural language understanding, particularly topic classification, sentiment analysis, question answering and language translation. We think that deep learning will have many more successes in the near future because it requires very little engineering by hand, so it can easily take advantage of increases in the amount of available computation and data. New learning algorithms and architectures that are currently being developed for deep neural networks will only accelerate this progress.

Computer Science
Deep Learning and Machine Learning -- Python Data Structures and Mathematics Fundamental: From Theory to Practice

Silin Chen,Ziqian Bi,Junyu Liu,Benji Peng,Sen Zhang,Xuanhe Pan,Jiawei Xu,Jinlang Wang,Keyu Chen,Caitlyn Heqi Yin,Pohsun Feng,Yizhu Wen,Tianyang Wang,Ming Li,Jintao Ren,Qian Niu,Ming Liu

2024-10-22

Abstract:This book provides a comprehensive introduction to the foundational concepts of machine learning (ML) and deep learning (DL). It bridges the gap between theoretical mathematics and practical application, focusing on Python as the primary programming language for implementing key algorithms and data structures. The book covers a wide range of topics, including basic and advanced Python programming, fundamental mathematical operations, matrix operations, linear algebra, and optimization techniques crucial for training ML and DL models. Advanced subjects like neural networks, optimization algorithms, and frequency domain methods are also explored, along with real-world applications of large language models (LLMs) and artificial intelligence (AI) in big data management. Designed for both beginners and advanced learners, the book emphasizes the critical role of mathematical principles in developing scalable AI solutions. Practical examples and Python code are provided throughout, ensuring readers gain hands-on experience in applying theoretical knowledge to solve complex problems in ML, DL, and big data analytics.

Machine Learning,Data Structures and Algorithms,Programming Languages
Deep Learning and Geometric Deep Learning: an introduction for mathematicians and physicists

R. Fioresi,F. Zanchetta

2023-05-10

Abstract:In this expository paper we want to give a brief introduction, with few key references for further reading, to the inner functioning of the new and successfull algorithms of Deep Learning and Geometric Deep Learning with a focus on Graph Neural Networks. We go over the key ingredients for these algorithms: the score and loss function and we explain the main steps for the training of a model. We do not aim to give a complete and exhaustive treatment, but we isolate few concepts to give a fast introduction to the subject. We provide some appendices to complement our treatment discussing Kullback-Leibler divergence, regression, Multi-layer Perceptrons and the Universal Approximation Theorem.

Machine Learning,Mathematical Physics
Mathematical theory of deep learning

Philipp Petersen,Jakob Zech

2024-10-11

Abstract:This book provides an introduction to the mathematical analysis of deep learning. It covers fundamental results in approximation theory, optimization theory, and statistical learning theory, which are the three main pillars of deep neural network theory. Serving as a guide for students and researchers in mathematics and related fields, the book aims to equip readers with foundational knowledge on the topic. It prioritizes simplicity over generality, and presents rigorous yet accessible results to help build an understanding of the essential mathematical concepts underpinning deep learning.

Machine Learning,History and Overview
The Modern Mathematics of Deep Learning

Julius Berner,Philipp Grohs,Gitta Kutyniok,Philipp Petersen

DOI: https://doi.org/10.48550/arXiv.2105.04026

IF: 5.414

2021-05-09

Machine Learning

Abstract:We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.
Deep architectures

Marco Gori,Alessandro Betti,Stefano Melacci

DOI: https://doi.org/10.1016/b978-0-32-389859-1.00012-x

IF: 5.414

2024-01-01

Machine Learning

Abstract:In this chapter, we give foundations on feedforward neural networks and incorporate some new recent developments on deep learning. This is a central topic in machine learning both for foundational and applicative implications. From the foundational side, it has been the subject of a multifacet analysis involving, among others, computational geometry, circuit theory, circuit complexity, approximation theory, optimization theory, and statistics. The impact that feedforward networks have been experiencing, however, is likely to be due mostly to their application in an impressive number of different domains.

computer science, artificial intelligence
Understanding of Machine Learning with Deep Learning: Architectures, Workflow, Applications and Future Directions

Mohammad Mustafa Taye

DOI: https://doi.org/10.3390/computers12050091

2023-04-25

Computers

Abstract:In recent years, deep learning (DL) has been the most popular computational approach in the field of machine learning (ML), achieving exceptional results on a variety of complex cognitive tasks, matching or even surpassing human performance. Deep learning technology, which grew out of artificial neural networks (ANN), has become a big deal in computing because it can learn from data. The ability to learn enormous volumes of data is one of the benefits of deep learning. In the past few years, the field of deep learning has grown quickly, and it has been used successfully in a wide range of traditional fields. In numerous disciplines, including cybersecurity, natural language processing, bioinformatics, robotics and control, and medical information processing, deep learning has outperformed well-known machine learning approaches. In order to provide a more ideal starting point from which to create a comprehensive understanding of deep learning, also, this article aims to provide a more detailed overview of the most significant facets of deep learning, including the most current developments in the field. Moreover, this paper discusses the significance of deep learning and the various deep learning techniques and networks. Additionally, it provides an overview of real-world application areas where deep learning techniques can be utilised. We conclude by identifying possible characteristics for future generations of deep learning modelling and providing research suggestions. On the same hand, this article intends to provide a comprehensive overview of deep learning modelling that can serve as a resource for academics and industry people alike. Lastly, we provide additional issues and recommended solutions to assist researchers in comprehending the existing research gaps. Various approaches, deep learning architectures, strategies, and applications are discussed in this work.
An introduction to deep learning for electromagnetics

Marco Salucci,Maokun Li,Xudong Chen,Andrea Massa

2022-01-01

Abstract:This chapter provided a gentle introduction to DL, recalling the most used terms in this field as well as describing some of the most widespread architectures in the recent literature and their applications in EM.
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

Michael M. Bronstein,Joan Bruna,Taco Cohen,Petar Veličković

2021-05-03

Abstract:The last decade has witnessed an experimental revolution in data science and machine learning, epitomised by deep learning methods. Indeed, many high-dimensional learning tasks previously thought to be beyond reach -- such as computer vision, playing Go, or protein folding -- are in fact feasible with appropriate computational scale. Remarkably, the essence of deep learning is built from two simple algorithmic principles: first, the notion of representation or feature learning, whereby adapted, often hierarchical, features capture the appropriate notion of regularity for each task, and second, learning by local gradient-descent type methods, typically implemented as backpropagation. While learning generic functions in high dimensions is a cursed estimation problem, most tasks of interest are not generic, and come with essential pre-defined regularities arising from the underlying low-dimensionality and structure of the physical world. This text is concerned with exposing these regularities through unified geometric principles that can be applied throughout a wide spectrum of applications. Such a 'geometric unification' endeavour, in the spirit of Felix Klein's Erlangen Program, serves a dual purpose: on one hand, it provides a common mathematical framework to study the most successful neural network architectures, such as CNNs, RNNs, GNNs, and Transformers. On the other hand, it gives a constructive procedure to incorporate prior physical knowledge into neural architectures and provide principled way to build future architectures yet to be invented.

Machine Learning,Artificial Intelligence,Computational Geometry,Computer Vision and Pattern Recognition
Deep Learning Model And Its Application In Big Data

Yuanming Zhou,Shifeng Zhao,Xuesong Wang,Wei Liu

DOI: https://doi.org/10.1007/978-3-319-91797-9_55

2018-01-01

Abstract:In the era of big data, many of the data that previously seemed hard to collect and use began to be utilized, resulting in an increase of millions of the data to be processed. In order to obtain valuable information, large-scale data can be processed and analyzed using a well-developed deep learning framework. This study introduces the concept of deep learning and three common deep learning models - Multilayer Perceptron, Convolutional Neural Network and Recurrent Neural Network, and analyzes the improvement of the model in dealing with large-scale data and gives the capacity and diversity analysis. Introducing the innovative application of deep learning in various fields under big data. Looking forward to the development of deep learning in the era of big data, the integration of big data and in-depth learning will make breakthroughs in various fields. Through constant innovation, they will gradually create more value for mankind.
Deep Learning and Machine Learning, Advancing Big Data Analytics and Management: Unveiling AI's Potential Through Tools, Techniques, and Applications

Pohsun Feng,Ziqian Bi,Yizhu Wen,Xuanhe Pan,Benji Peng,Ming Liu,Jiawei Xu,Keyu Chen,Junyu Liu,Caitlyn Heqi Yin,Sen Zhang,Jinlang Wang,Qian Niu,Ming Li,Tianyang Wang

2024-10-02

Abstract:This book serves as an introduction to deep learning and machine learning, focusing on their applications in big data analytics. It covers essential concepts, tools like ChatGPT and Claude, hardware recommendations, and practical guidance on setting up development environments using libraries like PyTorch and TensorFlow. Designed for beginners and advanced users alike, it provides step-by-step instructions, hands-on projects, and insights into AI's future, including AutoML and edge computing.

Computation and Language,Machine Learning
Artificial Neural Network and Deep Learning: Fundamentals and Theory

M. M. Hammad

2024-08-13

Abstract:"Artificial Neural Network and Deep Learning: Fundamentals and Theory" offers a comprehensive exploration of the foundational principles and advanced methodologies in neural networks and deep learning. This book begins with essential concepts in descriptive statistics and probability theory, laying a solid groundwork for understanding data and probability distributions. As the reader progresses, they are introduced to matrix calculus and gradient optimization, crucial for training and fine-tuning neural networks. The book delves into multilayer feed-forward neural networks, explaining their architecture, training processes, and the backpropagation algorithm. Key challenges in neural network optimization, such as activation function saturation, vanishing and exploding gradients, and weight initialization, are thoroughly discussed. The text covers various learning rate schedules and adaptive algorithms, providing strategies to optimize the training process. Techniques for generalization and hyperparameter tuning, including Bayesian optimization and Gaussian processes, are also presented to enhance model performance and prevent overfitting. Advanced activation functions are explored in detail, categorized into sigmoid-based, ReLU-based, ELU-based, miscellaneous, non-standard, and combined types. Each activation function is examined for its properties and applications, offering readers a deep understanding of their impact on neural network behavior. The final chapter introduces complex-valued neural networks, discussing complex numbers, functions, and visualizations, as well as complex calculus and backpropagation algorithms. This book equips readers with the knowledge and skills necessary to design, and optimize advanced neural network models, contributing to the ongoing advancements in artificial intelligence.

Machine Learning
DL 101: Basic introduction to deep learning with its application in biomedical related fields

Tianyu Zhan

DOI: https://doi.org/10.1002/sim.9564

2022-09-01

Statistics in Medicine

Abstract:Deep learning is a subfield of machine learning used to learn representations of data by successive layers. Remarkable achievements and breakthroughs have been made in image classification, speech recognition, et cetera, but the full capability of deep learning is still under exploration. As statistical researchers and practitioners, we are especially interested in leveraging and advancing deep learning techniques to address important and impactive problems in biomedical and other related fields. In this article, we provide a basic introduction to Feedforward Neural Networks (FNN) along with some intuitive explanations behind its strong functional representation. Guidance is provided on how to choose quite a few hyperparameters in neural networks for a specific problem. We further discuss several more advanced frameworks in deep learning. Some successful applications of deep learning in biomedical fields are also demonstrated. With this beginner's guide, we hope that interested readers can include deep learning in their toolbox to tackle future real‐world questions and challenges.

public, environmental & occupational health,medicine, research & experimental,medical informatics,mathematical & computational biology,statistics & probability
Fundamental Components of Deep Learning: A category-theoretic approach

Bruno Gavranović

2024-03-13

Abstract:Deep learning, despite its remarkable achievements, is still a young field. Like the early stages of many scientific disciplines, it is marked by the discovery of new phenomena, ad-hoc design decisions, and the lack of a uniform and compositional mathematical foundation. From the intricacies of the implementation of backpropagation, through a growing zoo of neural network architectures, to the new and poorly understood phenomena such as double descent, scaling laws or in-context learning, there are few unifying principles in deep learning. This thesis develops a novel mathematical foundation for deep learning based on the language of category theory. We develop a new framework that is a) end-to-end, b) unform, and c) not merely descriptive, but prescriptive, meaning it is amenable to direct implementation in programming languages with sufficient features. We also systematise many existing approaches, placing many existing constructions and concepts from the literature under the same umbrella. In Part I we identify and model two main properties of deep learning systems parametricity and bidirectionality by we expand on the previously defined construction of actegories and Para to study the former, and define weighted optics to study the latter. Combining them yields parametric weighted optics, a categorical model of artificial neural networks, and more. Part II justifies the abstractions from Part I, applying them to model backpropagation, architectures, and supervised learning. We provide a lens-theoretic axiomatisation of differentiation, covering not just smooth spaces, but discrete settings of boolean circuits as well. We survey existing, and develop new categorical models of neural network architectures. We formalise the notion of optimisers and lastly, combine all the existing concepts together, providing a uniform and compositional framework for supervised learning.

Machine Learning,Artificial Intelligence,Category Theory
An Introduction to Neural Networks and Deep Learning

Heung-Il Suk

DOI: https://doi.org/10.1016/b978-0-12-810408-8.00002-x

2017-01-01

Abstract:Artificial neural networks, conceptually and structurally inspired by neural systems, are of great interest along with deep learning, thanks to their great successes in various fields including medical imaging analysis. In this chapter, we describe the fundamental concepts and ideas of (deep) neural networks and explain algorithmic advances to learn network parameters efficiently by avoiding overfitting. Specifically, this chapter focuses on introducing (i) feed-forward neural networks, (ii) gradient descent-based parameter optimization algorithms, (iii) different types of deep models, (iv) technical tricks for fast and robust training of deep models, and (v) open source deep learning frameworks for quick practice.
Deep Learning in Deterministic Computational Mechanics

Leon Herrmann,Stefan Kollmannsberger

DOI: https://doi.org/10.48550/arXiv.2309.15421

2023-09-27

Abstract:The rapid growth of deep learning research, including within the field of computational mechanics, has resulted in an extensive and diverse body of literature. To help researchers identify key concepts and promising methodologies within this field, we provide an overview of deep learning in deterministic computational mechanics. Five main categories are identified and explored: simulation substitution, simulation enhancement, discretizations as neural networks, generative approaches, and deep reinforcement learning. This review focuses on deep learning methods rather than applications for computational mechanics, thereby enabling researchers to explore this field more effectively. As such, the review is not necessarily aimed at researchers with extensive knowledge of deep learning -- instead, the primary audience is researchers at the verge of entering this field or those who attempt to gain an overview of deep learning in computational mechanics. The discussed concepts are, therefore, explained as simple as possible.

Machine Learning
Deep Neural Networks: A Signal Processing Perspective

Heikki Huttunen

DOI: https://doi.org/10.1007/978-3-319-91734-4_4

2018-10-14

Abstract:Deep learning has rapidly become the state of the art in machine learning, surpassing traditional approaches by a significant margin for many widely studied benchmark sets. Although the basic structure of a deep neural network is very close to a traditional 1990s style network, a few novel components enable successful training of extremely deep networks, thus allowing a completely novel sphere of applications—often reaching human-level accuracy and beyond. Below, we familiarize the reader with the brief history of deep learning and discuss the most significant milestones over the years. We also describe the fundamental components of a modern deep neural networks and emphasize their close connection to the basic operations of signal processing, such as the convolution and the Fast Fourier Transform. We study the importance of pretraining with examples and, finally, we will discuss the real time deployment of a deep network; a topic often dismissed in textbooks; but increasingly important in future applications, such as self driving cars.

Deep Learning: An Introduction for Applied Mathematicians

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

Deep Learning: A Tutorial

Deep Learning

Deep Learning and Machine Learning -- Python Data Structures and Mathematics Fundamental: From Theory to Practice

Deep Learning and Geometric Deep Learning: an introduction for mathematicians and physicists

Mathematical theory of deep learning

The Modern Mathematics of Deep Learning

Deep architectures

Understanding of Machine Learning with Deep Learning: Architectures, Workflow, Applications and Future Directions

An introduction to deep learning for electromagnetics

Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

Deep Learning Model And Its Application In Big Data

Deep Learning and Machine Learning, Advancing Big Data Analytics and Management: Unveiling AI's Potential Through Tools, Techniques, and Applications

Artificial Neural Network and Deep Learning: Fundamentals and Theory

DL 101: Basic introduction to deep learning with its application in biomedical related fields

Fundamental Components of Deep Learning: A category-theoretic approach

An Introduction to Neural Networks and Deep Learning

Deep Learning in Deterministic Computational Mechanics

Deep Neural Networks: A Signal Processing Perspective