Abstract:Networks in machine learning offer examples of complex high-dimensional dynamical systems reminiscent of biological systems. Here, we study the learning dynamics of Generalized Hopfield networks, which permit a visualization of internal memories. These networks have been shown to proceed through a 'feature-to-prototype' transition, as the strength of network nonlinearity is increased, wherein the learned, or terminal, states of internal memories transition from mixed to pure states. Focusing on the prototype learning dynamics of the internal memories we observe a strong resemblance to the canalized, or low-dimensional, dynamics of cells as they differentiate within a Waddingtonian landscape. Dynamically, we demonstrate that learning in a Generalized Hopfield Network proceeds through sequential 'splits' in memory space. Furthermore, order of splitting is interpretable and reproducible. The dynamics between the splits are canalized in the Waddington sense -- robust to variations in detailed aspects of the system. In attempting to make the analogy a rigorous equivalence, we study smaller subsystems that exhibit similar properties to the full system. We combine analytical calculations with numerical simulations to study the dynamical emergence of the feature-to-prototype transition, and the behaviour of splits in the landscape, saddles points, visited during learning. We exhibit regimes where saddles appear and disappear through saddle-node bifurcations, qualitatively changing the distribution of learned memories as the strength of the nonlinearity is varied -- allowing us to systematically investigate the mechanisms that underlie the emergence of Waddingtonian dynamics. Memories can thus differentiate in a predictive and controlled way, revealing new bridges between experimental biology, dynamical systems theory, and machine learning.

Building Case-Based Preliminary Design Systems: A Hopfield Network Approach

The Hopfield Neural Networks Are Applied In The Conceptual Design For Example

An Integrated Cognitive Computing Approach for Systematic Conceptual Design

Application and Research on Hopfield in Computer Programming

Online Training of Hopfield Networks using Predictive Coding

A Feed-Forward Neural Network for Increasing the Hopfield-Network Storage Capacity

Pattern Storage and Hopfield Neural Associative Memory with Hidden Structure.

Learning associative memories by error backpropagation.

Hopfield Network Based Optimization of Mechanical Design

Prototype Analysis in Hopfield Networks with Hebbian Learning

Heteroclinic cycles in Hopfield networks

Some Exact Results of Hopfield Neural Networks and Applications

A New Approach to the Design of Hopfield Associative Memory

Analysis and Design of Asymmetric Hopfield Networks with Discrete-Time Dynamics

An Internal Mechanism for Detecting Parasite Attractors in a Hopfield Network

Simplicial Hopfield networks

Design And Analysis Of Positively Self-Feedbacked Hopfield Neural Network For Crossbar Switching

A New Kind of Hopfield Networks for Finding Global Optimum

Hopfield model with planted patterns: a teacher-student self-supervised learning model

A High Order Neural Network to Solve N-queens Problem

A Waddington landscape for prototype learning in generalized Hopfield networks