Introduction to linear algebra for neural networks

Edmund T. Rolls

2023-01-01

Abstract:Subject Cognition and Behavioural Neuroscience Collection: Oxford Scholarship Online

What problem does this paper attempt to address?

Six lectures on linearized neural networks

Theodor Misiakiewicz,Andrea Montanari

DOI: https://doi.org/10.1088/1742-5468/ad292a

2024-10-31

Journal of Statistical Mechanics Theory and Experiment

Abstract:This tutorial examines what can be learnt about the behavior of multi-layer neural networks from the analysis of linear models. While there are important gaps between neural networks and their linear counterparts, many useful lessons can be learnt by studying the latter. A few preliminary remarks, before diving into the math: • We will not assume specific background in machine learning, let alone neural networks. On the other hand, we will assume some graduate-level mathematics, in particular probability theory (however, we will refer to the literature for complete proofs.) • Some of the notations that are used throughout the text will be summarized in appendix A. • We will keep bibliographic references in the main text to a minimum. A short guide to the literature is given in appendix B.

physics, mathematical,mechanics
Neuronal encoding, and information theory

Edmund T. Rolls

2023-01-01

Abstract:Subject Cognition and Behavioural Neuroscience Collection: Oxford Scholarship Online
Linear Dynamics & Control of Brain Networks

Jason Z. Kim,Danielle S. Bassett

DOI: https://doi.org/10.48550/arXiv.1902.03309

2019-02-09

Abstract:The brain is an intricately structured organ responsible for the rich emergent dynamics that support the complex cognitive functions we enjoy as humans. With around $10^{11}$ neurons and $10^{15}$ synapses, understanding how the human brain works has proven to be a daunting endeavor, requiring concerted collaboration across traditional disciplinary boundaries. In some cases, that collaboration has occurred between experimentalists and technicians, who offer new physical tools to measure and manipulate neural function. In other contexts, that collaboration has occurred between experimentalists and theorists, who offer new conceptual tools to explain existing data and inform new directions for empirical research. In this chapter, we offer an example of the latter. Specifically, we focus on the simple but powerful framework of linear systems theory as a useful tool both for capturing biophysically relevant parameters of neural activity and connectivity, and for analytical and numerical study. We begin with a brief overview of state-space representations and linearization of neural models for non-linear dynamical systems. We then derive core concepts in the theory of linear systems such as the impulse and controlled responses to external stimuli, achieving desired state transitions, controllability, and minimum energy control. Afterwards, we discuss recent advances in the application of linear systems theory to structural and functional brain data across multiple spatial and temporal scales, along with methodological considerations and limitations. We close with a brief discussion of open frontiers and our vision for the future.

Neurons and Cognition,Optimization and Control,Biological Physics
Introduction to Machine Learning

Laurent Younes

2024-09-04

Abstract:This book introduces the mathematical foundations and techniques that lead to the development and analysis of many of the algorithms that are used in machine learning. It starts with an introductory chapter that describes notation used throughout the book and serve at a reminder of basic concepts in calculus, linear algebra and probability and also introduces some measure theoretic terminology, which can be used as a reading guide for the sections that use these tools. The introductory chapters also provide background material on matrix analysis and optimization. The latter chapter provides theoretical support to many algorithms that are used in the book, including stochastic gradient descent, proximal methods, etc. After discussing basic concepts for statistical prediction, the book includes an introduction to reproducing kernel theory and Hilbert space techniques, which are used in many places, before addressing the description of various algorithms for supervised statistical learning, including linear methods, support vector machines, decision trees, boosting, or neural networks. The subject then switches to generative methods, starting with a chapter that presents sampling methods and an introduction to the theory of Markov chains. The following chapter describe the theory of graphical models, an introduction to variational methods for models with latent variables, and to deep-learning based generative models. The next chapters focus on unsupervised learning methods, for clustering, factor analysis and manifold learning. The final chapter of the book is theory-oriented and discusses concentration inequalities and generalization bounds.

Machine Learning
A rigorous introduction to linear models

Jun Lu

2024-02-04

Abstract:This book is meant to provide an introduction to linear models and the theories behind them. Our goal is to give a rigorous introduction to the readers with prior exposure to ordinary least squares. In machine learning, the output is usually a nonlinear function of the input. Deep learning even aims to find a nonlinear dependence with many layers, which require a large amount of computation. However, most of these algorithms build upon simple linear models. We then describe linear models from different perspectives and find the properties and theories behind the models. The linear model is the main technique in regression problems, and the primary tool for it is the least squares approximation, which minimizes a sum of squared errors. This is a natural choice when we're interested in finding the regression function which minimizes the corresponding expected squared error. This book is primarily a summary of purpose, significance of important theories behind linear models, e.g., distribution theory and the minimum variance estimator. We first describe ordinary least squares from three different points of view, upon which we disturb the model with random noise and Gaussian noise. Through Gaussian noise, the model gives rise to the likelihood so that we introduce a maximum likelihood estimator. It also develops some distribution theories via this Gaussian disturbance. The distribution theory of least squares will help us answer various questions and introduce related applications. We then prove least squares is the best unbiased linear model in the sense of mean squared error, and most importantly, it actually approaches the theoretical limit. We end up with linear models with the Bayesian approach and beyond.

Machine Learning
Neural Networks and Brain Function

Edmund T. Rolls

DOI: https://doi.org/10.1093/acprof:oso/9780198524328.001.0001

2016-01-01

Abstract:If you really want to be smarter, reading can be one of the lots ways to evoke and realize. Many people who like reading will have more knowledge and experiences. Reading can be a way to gain information from economics, politics, science, fiction, literature, religion, and many others. As one of the part of book categories, neural networks and brain function always becomes the most wanted book. Many people are absolutely searching for this book. It means that many love to read this kind of book.
Machine learning with neural networks

B. Mehlig

DOI: https://doi.org/10.1017/9781108860604

2021-10-27

Abstract:These are lecture notes for a course on machine learning with neural networks for scientists and engineers that I have given at Gothenburg University and Chalmers Technical University in Gothenburg, Sweden. The material is organised into three parts: Hopfield networks, supervised learning of labeled data, and learning algorithms for unlabeled data sets. Part I introduces stochastic recurrent networks: Hopfield networks and Boltzmann machines. The analysis of their learning rules sets the scene for the later parts. Part II describes supervised learning with multilayer perceptrons and convolutional neural networks. This part starts with a simple geometrical interpretation of the learning rule and leads to the recent successes of convolutional networks in object recognition, recurrent networks in language processing, and reservoir computers in time-series analysis. Part III explains what neural networks can learn about data that is not labeled. This part begins with a description of unsupervised learning techniques for clustering of data, non-linear projections, and embeddings. A section on autoencoders explains how to learn without labels using convolutional networks, and the last chapter is dedicated to reinforcement learning. The overall goal of the course is to explain the fundamental principles that allow neural networks to learn, emphasising ideas and concepts that are common to all three parts. The present version does not contain exercises (copyright owned by Cambridge University Press). The complete book is available at <a class="link-external link-https" href="https://www.cambridge.org/gb/academic/subjects/physics/statistical-physics/machine-learning-neural-networks-introduction-scientists-and-engineers?format=HB" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Statistical Mechanics
An Introduction to Neural Information Processing

Peiji Liang,Si Wu,Fanji Gu

DOI: https://doi.org/10.1007/978-94-017-7393-5

2015-01-01

Abstract:This book provides an overview of neural information processing research, which is one of the most important branches of neuroscience today. Neural information processing is an interdisciplinary subject, and the merging interaction between neuroscience and mathematics, physics, as well as information science plays a key role in the development of this field. This book begins with the anatomy of the central nervous system, followed by an introduction to various information processing models at different levels. The authors all have extensive experience in mathematics, physics and biomedical engineering,and have worked in this multidisciplinary area for a number of years. They present classical examples of how the pioneers in this field used theoretical analysis, mathematical modeling and computer simulation to solve neurobiological problems, and share their experiences and lessons learned. The book is intended for researchers and students with a mathematics, physics or informatics background who are interested in brain research and keen to understand the necessary neurobiology and how they can use their specialties to address neurobiological problems. It is also provides inspiration for neuroscience students who are interested in learning how to use mathematics, physics or informatics approaches to solve problems in their field.
Understanding Neural Network Systems for Image Analysis using Vector Spaces and Inverse Maps

Rebecca Pattichis,Marios S. Pattichis

2024-02-01

Abstract:There is strong interest in developing mathematical methods that can be used to understand complex neural networks used in image analysis. In this paper, we introduce techniques from Linear Algebra to model neural network layers as maps between signal spaces. First, we demonstrate how signal spaces can be used to visualize weight spaces and convolutional layer kernels. We also demonstrate how residual vector spaces can be used to further visualize information lost at each layer. Second, we introduce the concept of invertible networks and an algorithm for computing input images that yield specific outputs. We demonstrate our approach on two invertible networks and ResNet18.

Computer Vision and Pattern Recognition,Machine Learning
A Gentle Introduction to Artificial Neural Networks

Zhongheng Zhang

DOI: https://doi.org/10.21037/atm.2016.06.20

IF: 3.616

2016-01-01

Annals of Translational Medicine

Abstract:Artificial neural network (ANN) is a flexible and powerful machine learning technique. However, it is under utilized in clinical medicine because of its technical challenges. The article introduces some basic ideas behind ANN and shows how to build ANN using R in a step-by-step framework. In topology and function, ANN is in analogue to the human brain. There are input and output signals transmitting from input to output nodes. Input signals are weighted before reaching output nodes according to their respective importance. Then the combined signal is processed by activation function. I simulated a simple example to illustrate how to build a simple ANN model using nnet() function. This function allows for one hidden layer with varying number of units in that layer. The basic structure of ANN can be visualized with plug-in plot.nnet() function. The plot function is powerful that it allows for varieties of adjustment to the appearance of the neural networks. Prediction with ANN can be performed with predict() function, similar to that of conventional generalized linear models. Finally, the prediction power of ANN is examined using confusion matrix and average accuracy. It appears that ANN is slightly better than conventional linear model.
Revisit the Fundamental Theorem of Linear Algebra

Jun Lu

DOI: https://doi.org/10.48550/arXiv.2108.04432

2021-08-10

Abstract:This survey is meant to provide an introduction to the fundamental theorem of linear algebra and the theories behind them. Our goal is to give a rigorous introduction to the readers with prior exposure to linear algebra. Specifically, we provide some details and proofs of some results from (Strang, 1993). We then describe the fundamental theorem of linear algebra from different views and find the properties and relationships behind the views. The fundamental theorem of linear algebra is essential in many fields, such as electrical engineering, computer science, machine learning, and deep learning. This survey is primarily a summary of purpose, significance of important theories behind it. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in theory behind the fundamental theorem of linear algebra and rigorous analysis in order to seamlessly introduce its properties in four subspaces in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results and given the paucity of scope to present this discussion, e.g., the separated analysis of the (orthogonal) projection matrices. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields. Some excellent examples include (Rose, 1982; Strang, 2009; Trefethen and Bau III, 1997; Strang, 2019, 2021).

Machine Learning,Numerical Analysis
Computational neuroscience of vision

Edmund T. Rolls,Gustavo Deco

DOI: https://doi.org/10.1093/acprof:oso/9780198524885.001.0001

2002-01-01

Abstract:Abstract This book presents the highly complex subject of vision, focusing on the visual information processing and computational operations in the visual system that lead to representations of objects in the brain. In addition to visual processing, it also considers how visual inputs reach and are involved in the computations underlying a wide range of behaviour, thus providing a foundation for understanding the operation of a number of different brain systems.
Linear Algebra In A Jiffy

Leo Livshits

2024-03-30

Abstract:Every $n$-tuple in $\mathbb{F}^{n}$ has a first non-zero entry and a last non-zero entry. What do the positions of such entries in the elements of a subspace W of $\mathbb{F}^{n}$ reveal about W? It turns out, a great deal! This insight offers a beeline to the fundamental results on general bases and dimension (including dimensions of complements, rank of a transpose, Rank-Nullity Theorem, Full Rank Factorization, and existence/uniqueness of RREF), bypassing some of the traditional stumbling blocks and time sinks.

History and Overview,Rings and Algebras
An Introduction to Neural Networks and Deep Learning

Heung-Il Suk

DOI: https://doi.org/10.1016/b978-0-12-810408-8.00002-x

2017-01-01

Abstract:Artificial neural networks, conceptually and structurally inspired by neural systems, are of great interest along with deep learning, thanks to their great successes in various fields including medical imaging analysis. In this chapter, we describe the fundamental concepts and ideas of (deep) neural networks and explain algorithmic advances to learn network parameters efficiently by avoiding overfitting. Specifically, this chapter focuses on introducing (i) feed-forward neural networks, (ii) gradient descent-based parameter optimization algorithms, (iii) different types of deep models, (iv) technical tricks for fast and robust training of deep models, and (v) open source deep learning frameworks for quick practice.
Tensor-Based Foundations of Ordinary Least Squares and Neural Network Regression Models

Roberto Dias Algarte

2024-11-20

Abstract:This article introduces a novel approach to the mathematical development of Ordinary Least Squares and Neural Network regression models, diverging from traditional methods in current Machine Learning literature. By leveraging Tensor Analysis and fundamental matrix computations, the theoretical foundations of both models are meticulously detailed and extended to their complete algorithmic forms. The study culminates in the presentation of three algorithms, including a streamlined version of the Backpropagation Algorithm for Neural Networks, illustrating the benefits of this new mathematical approach.

Machine Learning
Beyond Linear Algebra

Bernd Sturmfels

DOI: https://doi.org/10.48550/arXiv.2108.09494

2021-08-21

Abstract:Our title challenges the reader to venture beyond linear algebra in designing models and in thinking about numerical algorithms for identifying solutions. This article accompanies the author's lecture at the International Congress of Mathematicians 2022. It covers recent advances in the study of critical point equations in optimization and statistics, and it explores the role of nonlinear algebra in the study of linear PDE with constant coefficients.

Algebraic Geometry,Symbolic Computation,Optimization and Control,Statistics Theory
Neural Networks, Hypersurfaces, and Radon Transforms

Soheil Kolouri,Xuwang Yin,Gustavo K. Rohde

DOI: https://doi.org/10.48550/arXiv.1907.02220

2019-07-04

Abstract:Connections between integration along hypersufaces, Radon transforms, and neural networks are exploited to highlight an integral geometric mathematical interpretation of neural networks. By analyzing the properties of neural networks as operators on probability distributions for observed data, we show that the distribution of outputs for any node in a neural network can be interpreted as a nonlinear projection along hypersurfaces defined by level surfaces over the input data space. We utilize these descriptions to provide new interpretation for phenomena such as nonlinearity, pooling, activation functions, and adversarial examples in neural network-based learning problems.

Machine Learning
Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning

Nadav Cohen,Noam Razin

2024-08-25

Abstract:These notes are based on a lecture delivered by NC on March 2021, as part of an advanced course in Princeton University on the mathematical understanding of deep learning. They present a theory (developed by NC, NR and collaborators) of linear neural networks -- a fundamental model in the study of optimization and generalization in deep learning. Practical applications born from the presented theory are also discussed. The theory is based on mathematical tools that are dynamical in nature. It showcases the potential of such tools to push the envelope of our understanding of optimization and generalization in deep learning. The text assumes familiarity with the basics of statistical learning theory. Exercises (without solutions) are included.

Machine Learning,Artificial Intelligence
Some Theoretical Estimations of Spatial Distribution of Compton Backscattered Laser Photons Beam

Yu.P. Peresunko

DOI: https://doi.org/10.48550/arXiv.physics/0308031

2003-08-09

Abstract:Spatial distribution of intensity, degree and direction of linear polarization of tagged photon beam, which is obtained due to Compton backscattering of laser light on high-energy electron beam, are calculated. Effects of angular dispersion and spatial spread of electron beam are taken into account. Calculations have been carried out for the example of LEGS facility.

Instrumentation and Detectors,Accelerator Physics
Nonlinear and Nonparametric Extensions

René Vidal,Yi Ma,S. Shankar Sastry

DOI: https://doi.org/10.1007/978-0-387-87811-9_4

2016-01-01

Abstract:In the previous chapters, we studied the problem of fitting a low-dimensional linear or affine subspace to a collection of points. In practical applications, however, a linear or affine subspace may not be able to capture nonlinear structures in the data. For instance, consider the set of all images of a face obtained by rotating it about its main axis of symmetry. While all such images live in a high-dimensional space whose dimension is the number of pixels, there is only one degree of freedom in the data, namely the angle of rotation. In fact, the space of all such images is a one-dimensional circle embedded in a high-dimensional space, whose structure is not well captured by a one-dimensional line. More generally, a collection of face images observed from different viewpoints is not well approximated by a single linear or affine subspace, as illustrated in the following example.

Introduction to linear algebra for neural networks

Six lectures on linearized neural networks

Neuronal encoding, and information theory

Linear Dynamics & Control of Brain Networks

Introduction to Machine Learning

A rigorous introduction to linear models

Neural Networks and Brain Function

Machine learning with neural networks

An Introduction to Neural Information Processing

Understanding Neural Network Systems for Image Analysis using Vector Spaces and Inverse Maps

A Gentle Introduction to Artificial Neural Networks

Revisit the Fundamental Theorem of Linear Algebra

Computational neuroscience of vision

Linear Algebra In A Jiffy

An Introduction to Neural Networks and Deep Learning

Tensor-Based Foundations of Ordinary Least Squares and Neural Network Regression Models

Beyond Linear Algebra

Neural Networks, Hypersurfaces, and Radon Transforms

Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning

Some Theoretical Estimations of Spatial Distribution of Compton Backscattered Laser Photons Beam

Nonlinear and Nonparametric Extensions