Abstract:The performance of artificial neural networks (ANNs) degrades when training data are limited or imbalanced. In contrast, the human brain can learn quickly from just a few examples. Here, we investigated the role of sleep in improving the performance of ANNs trained with limited data on the MNIST and Fashion MNIST datasets. Sleep was implemented as an unsupervised phase with local Hebbian type learning rules. We found a significant boost in accuracy after the sleep phase for models trained with limited data in the range of 0.5-10% of total MNIST or Fashion MNIST datasets. When more than 10% of the total data was used, sleep alone had a slight negative impact on performance, but this was remedied by fine-tuning on the original data. This study sheds light on a potential synaptic weight dynamics strategy employed by the brain during sleep to enhance memory performance when training data are limited or imbalanced.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: when the training data is limited or unbalanced, the performance of artificial neural networks (ANNs) will decline significantly. Specifically: 1. **Limited data problem**: In deep learning, existing techniques usually require a large amount of training data to achieve good performance. However, in practical applications, sometimes only a small amount of labeled data can be obtained, which makes it difficult for the model to generalize and is prone to over - fitting. 2. **Data imbalance problem**: The number of samples in some categories in the training dataset is much larger than that in other categories. This imbalance will lead to the model's poor prediction ability for the minority categories, thereby reducing the overall accuracy. To solve these problems, the author draws on the sleep mechanism in biology, especially the role of sleep in memory consolidation. They propose a method that simulates the sleep process - **unsupervised replay**, and adjust the weights of the neural network through local Hebbian - type learning rules, in order to improve the model performance in the case of limited or unbalanced data. ### Main contributions of the paper - **Introducing Sleep Phase**: After the training phase, the memory replay and synaptic plasticity during sleep are simulated in an unsupervised manner, which enhances the memory representation of the model. - **Improving performance on small datasets**: The experimental results show that when training with 0.5% to 10% of the MNIST or Fashion MNIST datasets, after the Sleep Phase treatment, the accuracy of the model is significantly improved (20 - 30%). - **Alleviating the impact of data imbalance**: For unbalanced datasets, the Sleep Phase can effectively improve the classification accuracy of under - represented categories while maintaining high accuracy for common categories. - **Applicable to trained models**: This method can be applied to already - trained models without adding additional data to further optimize their performance. ### Formula explanation The Hebbian - type learning rule mentioned in the paper can be expressed as: \[ \Delta w_{ij} = \eta \cdot (x_i - \bar{x}_i) \cdot (y_j - \bar{y}_j) \] where: - \( w_{ij} \) is the connection weight from neuron \( i \) to neuron \( j \). - \( x_i \) and \( y_j \) are the activation values of the feed - forward layer and the feedback layer respectively. - \( \bar{x}_i \) and \( \bar{y}_j \) are the corresponding average activation values. - \( \eta \) is the learning rate. This formula describes that if the feed - forward activation \( x_i \) follows the feedback activation \( y_j \), the connection between them is enhanced; otherwise, the connection is weakened. In conclusion, this research provides a new strategy to deal with the problems of limited and unbalanced data in deep learning by simulating the memory consolidation mechanism in biological sleep processes.

Sleep-Like Unsupervised Replay Improves Performance when Data are Limited or Unbalanced