Abstract:The dreaming Hopfield model constitutes a generalization of the Hebbian paradigm for neural networks, that is able to perform on-line learning when "awake" and also to account for off-line "sleeping" mechanisms. The latter have been shown to enhance storing in such a way that, in the long sleep-time limit, this model can reach the maximal storage capacity achievable by networks equipped with symmetric pairwise interactions. In this paper, we inspect the minimal amount of information that must be supplied to such a network to guarantee a successful generalization, and we test it both on random synthetic and on standard structured datasets (i.e., MNIST, Fashion-MNIST and Olivetti). By comparing these minimal thresholds of information with those required by the standard (i.e., always "awake") Hopfield model, we prove that the present network can save up to ∼90% of the dataset size, yet preserving the same performance of the standard counterpart. This suggests that sleep may play a pivotal role in explaining the gap between the large volumes of data required to train artificial neural networks and the relatively small volumes needed by their biological counterparts. Further, we prove that the model Cost function (typically used in statistical mechanics) admits a representation in terms of a standard Loss function (typically used in machine learning) and this allows us to analyze its emergent computational skills both theoretically and computationally: a quantitative picture of its capabilities as a function of its control parameters is achieved and consistency between the two approaches is highlighted. The resulting network is an associative memory for pattern recognition tasks that learns from examples on-line, generalizes correctly (in suitable regions of its control parameters) and optimizes its storage capacity by off-line sleeping: such a reduction of the training cost can be inspiring toward sustainable AI and in situations where data are relatively sparse.

The overfitted brain: Dreams evolved to assist generalization

Dreaming is All You Need

Continual learning benefits from multiple sleep mechanisms: NREM, REM, and Synaptic Downscaling

Hebbian dreaming for small datasets

Exploring the role of dreams: insights from recent studies

On the utility of dreaming: A general model for how learning in artificial agents can benefit from data hallucination

Biologically inspired sleep algorithm for artificial neural networks

Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones

Dreaming and the default network: A review, synthesis, and counterintuitive research proposal

Learning cortical representations through perturbed and adversarial dreaming

Learning beyond sensations: how dreams organize neuronal representations

Dreams, endocannabinoids and itinerant dynamics in neural networks: re elaborating Crick-Mitchison unlearning hypothesis

Do Agents Dream of Electric Sheep?: Improving Generalization in Reinforcement Learning through Generative Learning

A model of autonomous interactions between hippocampus and neocortex driving sleep-dependent memory consolidation

A Study of Biologically Plausible Neural Network: The Role and Interactions of Brain-Inspired Mechanisms in Continual Learning

Sleep-Like Unsupervised Replay Improves Performance when Data are Limited or Unbalanced

Nightmares: a new neurocognitive model

Towards biologically plausible Dreaming and Planning in recurrent spiking networks

The neurocognition of dreaming: key questions and foci

Reduced and Redundant: Information Processing of Prediction Errors during Sleep

Dropout's Dream Land: Generalization from Learned Simulators to Reality