Abstract:Animal behaviour alternates between stochastic exploration and goal-directed actions, which are generated by the underlying neural dynamics. Previously, we demonstrated that the compositional Restricted Boltzmann Machine (cRBM) can decompose whole-brain activity of larval zebrafish data at the neural level into a small number ($\sim$100-200) of assemblies that can account for the stochasticity of the neural activity (van der Plas et al., eLife, 2023). Here we advance this representation by extending to a combined stochastic-dynamical representation to account for both aspects using the Recurrent Temporal RBM (RTRBM) and transfer-learning based on the cRBM estimate. We demonstrate that the functional advantage of the RTRBM is captured in the temporal weights on the hidden units, representing neural assemblies, both in simulated and experimental data. Our results show that the temporal expansion outperforms the stochastic-only cRBM in terms of generalization error and achieves more accurate representation of the moments in time. Lastly, we demonstrate that we can identify the original time-scale of assembly dynamics, by estimating multiple RTRBMs at different temporal resolutions. Together, we propose that RTRBMs are a valuable tool for capturing the combined stochastic and time-predictive dynamics of large-scale data sets.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to simultaneously capture the dynamic characteristics of random exploration and goal - directed behavior in neural activities. Specifically, the researchers hope to use the Recurrent Temporal Restricted Boltzmann Machine (RTRBM) to extend the previous method of decomposing the whole - brain activity data of zebrafish larvae using the compositional Restricted Boltzmann Machine (cRBM). In this way, they aim to establish a comprehensive model that can represent the randomness of neural activities and capture their temporal predictability. ### Research Background and Motivation 1. **The Concept of Neural Assemblies**: When a large number of neurons exhibit joint activities, they are usually considered to form a functional unit, namely a neural assembly. These assemblies are regarded as the basic computational units for performing cognitive functions such as short - term memory, sensorimotor computation, and decision - making. 2. **Advances in Neuroimaging Technologies**: In recent years, the development of neuroimaging technologies such as light - sheet microscopy has made it possible to study the roles of these neural assemblies in the brain. These technologies can record the functional activities of the entire brain volume, thereby studying how complex computations emerge in the brain. 3. **Challenges**: Extracting neural activation patterns from data sets containing about 100,000 or more neurons is a computational challenge. 4. **Previous Work**: Previous studies have used cRBM to identify neural assemblies in large - scale neural data, but cRBM can only capture static dependencies and cannot specifically explain temporal dependencies. ### Solution To overcome the above limitations, this paper proposes the use of the RTRBM model. RTRBM directly incorporates temporal dependencies into the model by introducing temporal dynamic characteristics. This method not only retains the advantages of cRBM (such as sparse connections) but also can better capture the temporal predictability of neural activities. ### Experimental Verification 1. **Simulated Data**: First, the researchers verified the effectiveness of RTRBM on simulated data, demonstrating its advantages in capturing temporal dependencies, especially performing better in the statistics of time - shifted data. 2. **Zebrafish Data**: Then, they applied RTRBM to the whole - brain activity data of zebrafish larvae, initializing the weights of RTRBM by the transfer learning method and training the model to capture the temporal dynamics of neural assemblies. The results show that RTRBM is superior to cRBM in multiple aspects, especially in reconstruction error and the accuracy of time - shifted statistics. ### Conclusion By introducing temporal dynamic characteristics, RTRBM can more accurately represent the randomness and temporal predictability of neural activities, thus providing a valuable tool for understanding and analyzing large - scale neural data. This method not only improves the interpretability of the model but also provides new directions for future research.

The Recurrent Temporal Restricted Boltzmann Machine Captures Neural Assembly Dynamics in Whole-brain Activity

The recurrent temporal restricted boltzmann machine

Learnable latent embeddings for joint behavioural and neural analysis

Brain dynamics uncovered using a machine-learning algorithm

Identification of Recurrent Dynamics in Distributed Neural Populations

Sequence Classification Restricted Boltzmann Machines With Gated Units

Ensemble perspective for understanding temporal credit assignment

Deep Neural Networks for Exploration of Transcriptome of Adult Mouse Brain

Linking Brain and Behavior States in Zebrafish Larvae Locomotion using Hidden Markov Models

Rapid, interpretable, data-driven models of neural dynamics using Recurrent Mechanistic Models

CrEIMBO: Cross Ensemble Interactions in Multi-view Brain Observations

Structural Credit Assignment with Coordinated Exploration

Extracting low-dimensional dynamics from multiple large-scale neural population recordings by learning to predict correlations

Linear-Time Sequence Classification using Restricted Boltzmann Machines

Learning spatiotemporal signals using a recurrent spiking network that discretizes time

Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution

A Markovian neural barcode representing mesoscale cortical spatiotemporal dynamics.

Restricted Boltzmann Machine with Adaptive Local Hidden Units.

TAVRNN: Temporal Attention-enhanced Variational Graph RNN Captures Neural Dynamics and Behavior

Investigating the temporal dynamics of electroencephalogram (EEG) microstates using recurrent neural networks

Deep Representations for Time-varying Brain Datasets