Abstract:Deep learning, despite its remarkable achievements, is still a young field. Like the early stages of many scientific disciplines, it is marked by the discovery of new phenomena, ad-hoc design decisions, and the lack of a uniform and compositional mathematical foundation. From the intricacies of the implementation of backpropagation, through a growing zoo of neural network architectures, to the new and poorly understood phenomena such as double descent, scaling laws or in-context learning, there are few unifying principles in deep learning. This thesis develops a novel mathematical foundation for deep learning based on the language of category theory. We develop a new framework that is a) end-to-end, b) unform, and c) not merely descriptive, but prescriptive, meaning it is amenable to direct implementation in programming languages with sufficient features. We also systematise many existing approaches, placing many existing constructions and concepts from the literature under the same umbrella. In Part I we identify and model two main properties of deep learning systems parametricity and bidirectionality by we expand on the previously defined construction of actegories and Para to study the former, and define weighted optics to study the latter. Combining them yields parametric weighted optics, a categorical model of artificial neural networks, and more. Part II justifies the abstractions from Part I, applying them to model backpropagation, architectures, and supervised learning. We provide a lens-theoretic axiomatisation of differentiation, covering not just smooth spaces, but discrete settings of boolean circuits as well. We survey existing, and develop new categorical models of neural network architectures. We formalise the notion of optimisers and lastly, combine all the existing concepts together, providing a uniform and compositional framework for supervised learning.

Deep Learning Framework From Scratch Using Numpy

A Framework for Distributed Deep Learning Layer Design in Python

Fundamental Components of Deep Learning: A category-theoretic approach

Deep Learning Applications with Python

Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

Monadic Deep Learning

Practical Aspects on Solving Differential Equations Using Deep Learning: A Primer

Deep Learning: An Introduction for Applied Mathematicians

Deep Learning and Machine Learning -- Python Data Structures and Mathematics Fundamental: From Theory to Practice

Deep Learning: From Basics to Building Deep Neural Networks with Python

Deep learning for gradient flows using the Brezis-Ekeland principle

Deep Learning: A Tutorial

Dive into Deep Learning

DeepPhysics: a physics aware deep learning framework for real-time simulation

Shared Data and Algorithms for Deep Learning in Fundamental Physics

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

Deep Teaching: Materials for Teaching Machine and Deep Learning

An Orchestrated Empirical Study on Deep Learning Frameworks and Platforms

Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond

An Analytic Layer-wise Deep Learning Framework with Applications to Robotics

Deep learning: a statistical viewpoint