Abstract:Based on the numerical manifold method principle, we developed a mathematical framework of a neural network manifold: Deep Manifold and discovered that neural networks: 1) is numerical computation combining forward and inverse; 2) have near infinite degrees of freedom; 3) exponential learning capacity with depth; 4) have self-progressing boundary conditions; 5) has training hidden bottleneck. We also define two concepts: neural network learning space and deep manifold space and introduce two concepts: neural network intrinsic pathway and fixed point. We raise three fundamental questions: 1). What is the training completion definition; 2). where is the deep learning convergence point (neural network fixed point); 3). How important is token timestamp in training data given negative time is critical in inverse problem.

What problem does this paper attempt to address?

This paper attempts to solve several core problems of neural networks in deep learning, specifically including: 1. **Defining the criteria for completion of training**: How to determine when a neural network has completed training is an open question. Currently, most training processes rely on the change of the loss value with respect to the labeled data to judge whether the training is completed, but this method is not always accurate or effective. 2. **Finding the convergence point in deep learning**: That is, the fixed point of the neural network. The paper explores whether the neural network will reach a stable fixed point during the training process, and the significance and properties of this fixed point. 3. **The importance of timestamps in training data**: Especially when dealing with inverse problems, how the concept of negative time (i.e., past time) is reflected in the training data, and the impact on model performance. The paper points out that the current training data lacks explicit timestamps, which may affect the model's understanding and processing of the time dimension. ### Main contributions of the paper - **Proposing the concept of "Neural Network Manifold"**: Based on the Numerical Manifold Method, the paper develops a mathematical framework to describe the manifold characteristics of neural networks, revealing the computational characteristics of neural networks in forward and backward propagation. - **Defining two concepts**: Neural Network Learning Space and Deep Manifold Space, and introducing the concepts of Neural Network Intrinsic Pathway and Fixed Point. - **Discussing the handling of high - order nonlinearity**: The paper analyzes in detail the capabilities and challenges of neural networks in dealing with high - order nonlinear problems, and proposes to alleviate the bottlenecks brought by high - order nonlinearity through techniques such as normalization, dropout, and skip - connections. - **Proposing new training and convergence theories**: The paper proposes a new perspective to understand the training process of neural networks, especially how to define the criteria for completion of training in the absence of a clear convergence point. ### Formula examples - **High - order nonlinear function**: \[ f=\sum_{j = 0}^{J}(\alpha_jx^j) \] When \(J>4\), there is no general closed - form solution, but it can be solved by numerical methods. - **Dynamic calculation of neural networks**: \[ \theta = F^{-1}(d)\approx\sum_{n = 1}^{\text{nodes}}f_n(x_n) \] where \(f_n\) is an integrable function and \(x\) is the input value. - **Exponential computing power**: \[ T\rightarrow O(\lambda^{W\cdot H}) \] where \(W\) is the number of nodes in each layer, \(H\) is the number of hidden layers, and \(\lambda\) is a constant. - **Learning transformation**: \[ T\rightarrow O(\lambda^{W\cdot H}):R_{\text{LS}}\cdot J\rightarrow R_{\text{dm}}\cdot J \] These formulas and concepts together form the core content of the paper, aiming to deeply understand and optimize the training and inference processes of neural networks.

Deep Manifold Part 1: Anatomy of Neural Network Manifold

Deep Manifold Transformation for Dimension Reduction

Deep Manifold Computing and Visualization

Deep Manifold Transformation for Nonlinear Dimensionality Reduction

Deep Manifold Computing and Visualization Using Elastic Locally Isometric Smoothness

Markov-Lipschitz Deep Learning

Exploring the Manifold of Neural Networks Using Diffusion Geometry

ManifoldNet: A Deep Network Framework for Manifold-valued Data

A Convergence Rate for Manifold Neural Networks

Matrix Manifold Neural Networks++

The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold

Deep neural networks architectures from the perspective of manifold learning

Construction of neural networks for realization of localized deep learning

Probing Biological and Artificial Neural Networks with Task-dependent Neural Manifolds

The Local Dimension of Deep Manifold.

For Manifold Learning, Deep Neural Networks can be Locality Sensitive Hash Functions

Deep nets for local manifold learning

Manifold Neural Network With Non-Gradient Optimization

High-dimensional manifold of solutions in neural networks: insights from statistical physics

Neural manifold analysis of brain circuit dynamics in health and disease