Abstract:Understanding neural dynamics is a central topic in machine learning, non-linear physics and neuroscience. However, the dynamics is non-linear, stochastic and particularly non-gradient, i.e., the driving force can not be written as gradient of a potential. These features make analytic studies very challenging. The common tool is the path integral approach or dynamical mean-field theory, but the drawback is that one has to solve the integro-differential or dynamical mean-field equations, which is computationally expensive and has no closed form solutions in general. From the aspect of associated Fokker-Planck equation, the steady state solution is generally unknown. Here, we treat searching for the steady states as an optimization problem, and construct an approximate potential related to the speed of the dynamics, and find that searching for the ground state of this potential is equivalent to running an approximate stochastic gradient dynamics or Langevin dynamics. Only in the zero temperature limit, the distribution of the original steady states can be achieved. The resultant stationary state of the dynamics follows exactly the canonical Boltzmann measure. Within this framework, the quenched disorder intrinsic in the neural networks can be averaged out by applying the replica method, which leads naturally to order parameters for the non-equilibrium steady states. Our theory reproduces the well-known result of edge-of-chaos, and further the order parameters characterizing the continuous transition are derived, and the order parameters are explained as fluctuations and responses of the steady states. Our method thus opens the door to analytically study the steady state landscape of the deterministic or stochastic high dimensional dynamics.

On the Mathematical Clarification of the SNAP-Back-Repeller in High-Dimensional Systems and Chaos in a Discrete Neural Network Model.

Snap-back Repeller and Chaos in Transiently Chaotic Neural Network

Generalized Snap-Back Repeller and Semi-Conjugacy to Shift Operators of Piecewise Continuous Transformations

Heteroclinical Repellers Imply Chaos

Coexisting Chaotic Attractors in A Single Neuron Model with Adapting Feedback Synapse

Impulsive Control of Stochastic Systems with Applications in Chaos Control, Chaos Synchronization, and Neural Networks

CONTROLLING CHAOS IN A CHAOTIC NEURON MODEL

Complex Dynamics of Noise-Perturbed Excitatory-Inhibitory Neural Networks With Intra-Correlative and Inter-Independent Connections

Nonlinear Dynamics Analysis of a Self-Organizing Recurrent Neural Network: Chaos Waning

Discontinuous transition to chaos in a canonical random neural network

Maximum Hyperchaos in Chaotic Nonmonotonic Neuronal Networks

Widespread neuronal chaos induced by slow oscillating currents

Inference and Chaos by a Network of Non-monotonic Neurons

Network-Induced Chaos in Integrate-And-Fire Neuronal Ensembles

The Chaos Theory and Its Application in Neural Network Modeling

A Model of Chaotic Neural Network with Applications in Optimization

An optimization-based equilibrium measure describes fixed points of non-equilibrium dynamics: application to edge of chaos

Route to chaos in a branching model of neural network dynamics

Chaos and firing patterns in a discrete fractional Hopfield neural network model

Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations

An optimization-based equilibrium measure describes non-equilibrium steady state dynamics: application to edge of chaos