Abstract:Networks in machine learning offer examples of complex high-dimensional dynamical systems reminiscent of biological systems. Here, we study the learning dynamics of Generalized Hopfield networks, which permit a visualization of internal memories. These networks have been shown to proceed through a 'feature-to-prototype' transition, as the strength of network nonlinearity is increased, wherein the learned, or terminal, states of internal memories transition from mixed to pure states. Focusing on the prototype learning dynamics of the internal memories we observe a strong resemblance to the canalized, or low-dimensional, dynamics of cells as they differentiate within a Waddingtonian landscape. Dynamically, we demonstrate that learning in a Generalized Hopfield Network proceeds through sequential 'splits' in memory space. Furthermore, order of splitting is interpretable and reproducible. The dynamics between the splits are canalized in the Waddington sense -- robust to variations in detailed aspects of the system. In attempting to make the analogy a rigorous equivalence, we study smaller subsystems that exhibit similar properties to the full system. We combine analytical calculations with numerical simulations to study the dynamical emergence of the feature-to-prototype transition, and the behaviour of splits in the landscape, saddles points, visited during learning. We exhibit regimes where saddles appear and disappear through saddle-node bifurcations, qualitatively changing the distribution of learned memories as the strength of the nonlinearity is varied -- allowing us to systematically investigate the mechanisms that underlie the emergence of Waddingtonian dynamics. Memories can thus differentiate in a predictive and controlled way, revealing new bridges between experimental biology, dynamical systems theory, and machine learning.

An Internal Mechanism for Detecting Parasite Attractors in a Hopfield Network

A Waddington landscape for prototype learning in generalized Hopfield networks

Prototype Analysis in Hopfield Networks with Hebbian Learning

Building Case-Based Preliminary Design Systems: A Hopfield Network Approach

Vector Symbolic Finite State Machines in Attractor Neural Networks

Hybrid Detector Set

New Distinguishing Method of Prototype Stability about Associative Hopfield Neural Networks

Hopfield Neural Network Multi-User Detection Based on Immune Algorithm

A Biological Immunity-Based Neuro Prototype for Few-Shot Anomaly Detection with Character Embedding

Neural Information Processing in Hierarchical Prototypical Networks

Dense Hopfield Networks in the Teacher-Student Setting

Hopfield-based Parallel Inference-counteracting multi-user Detection Algorithm GMHNN

Online Training of Hopfield Networks using Predictive Coding

Heteroclinic cycles in Hopfield networks

The Hopfield Neural Networks Are Applied In The Conceptual Design For Example

A New Approach to the Design of Hopfield Associative Memory

Vector-product Hopfield model

Prototype-based interpretation of the functionality of neurons in winner-take-all neural networks

A Feed-Forward Neural Network for Increasing the Hopfield-Network Storage Capacity

Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks