Abstract:Finding latent structures in data is drawing increasing attention in diverse fields such as image and signal processing, fluid dynamics, and machine learning. In this work we examine the problem of finding the main modes of gradient flows. Gradient descent is a fundamental process in optimization where its stochastic version is prominent in training of neural networks. Here our aim is to establish a consistent theory for gradient flows $\psi_t = P(\psi)$, where $P$ is a nonlinear homogeneous operator. Our proposed framework stems from analytic solutions of homogeneous flows, previously formalized by Cohen-Gilboa, where the initial condition $\psi_0$ admits the nonlinear eigenvalue problem $P(\psi_0)=\lambda \psi_0 $. We first present an analytic solution for \ac{DMD} in such cases. We show an inherent flaw of \ac{DMD}, which is unable to recover the essential dynamics of the flow. It is evident that \ac{DMD} is best suited for homogeneous flows of degree one. We propose an adaptive time sampling scheme and show its dynamics are analogue to homogeneous flows of degree one with a fixed step size. Moreover, we adapt \ac{DMD} to yield a real spectrum, using symmetric matrices. Our analytic solution of the proposed scheme recovers the dynamics perfectly and yields zero error. We then proceed to show that in the general case the orthogonal modes $\{ \phi_i \}$ are approximately nonlinear eigenfunctions $P(\phi_i) \approx\lambda_i \phi_i $. We formulate Orthogonal Nonlinear Spectral decomposition (\emph{OrthoNS}), which recovers the essential latent structures of the gradient descent process. Definitions for spectrum and filtering are given, and a Parseval-type identity is shown.

Dynamical Behaviors of the Gradient Flows for In-Context Learning

Absence of Closed-Form Descriptions for Gradient Flow in Two-Layer Narrow Networks

Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality

Gradient flow in parameter space is equivalent to linear interpolation in output space

Modes of Homogeneous Gradient Flows

In-context Learning and Gradient Descent Revisited

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

On Learning Gaussian Multi-index Models with Gradient Flow

Adversarial flows: A gradient flow characterization of adversarial attacks

Discretized Gradient Flow for Manifold Learning in the Space of Embeddings

Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows

Learning Thermodynamically Stable and Galilean Invariant Partial Differential Equations for Non-Equilibrium Flows

The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program

Finite Sample Analysis and Bounds of Generalization Error of Gradient Descent in In-Context Linear Regression

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Learning Time-Scales in Two-Layers Neural Networks

Saddle-to-saddle dynamics in diagonal linear networks*

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent

Kernel Approximation of Fisher-Rao Gradient Flows