Abstract:Objective.Learning dynamical latent state models for multimodal spiking and field potential activity can reveal their collective low-dimensional dynamics and enable better decoding of behavior through multimodal fusion. Toward this goal, developing unsupervised learning methods that are computationally efficient is important, especially for real-time learning applications such as brain-machine interfaces (BMIs). However, efficient learning remains elusive for multimodal spike-field data due to their heterogeneous discrete-continuous distributions and different timescales.Approach.Here, we develop a multiscale subspace identification (multiscale SID) algorithm that enables computationally efficient learning for modeling and dimensionality reduction for multimodal discrete-continuous spike-field data. We describe the spike-field activity as combined Poisson and Gaussian observations, for which we derive a new analytical SID method. Importantly, we also introduce a novel constrained optimization approach to learn valid noise statistics, which is critical for multimodal statistical inference of the latent state, neural activity, and behavior. We validate the method using numerical simulations and with spiking and local field potential population activity recorded during a naturalistic reach and grasp behavior.Main results.We find that multiscale SID accurately learned dynamical models of spike-field signals and extracted low-dimensional dynamics from these multimodal signals. Further, it fused multimodal information, thus better identifying the dynamical modes and predicting behavior compared to using a single modality. Finally, compared to existing multiscale expectation-maximization learning for Poisson-Gaussian observations, multiscale SID had a much lower training time while being better in identifying the dynamical modes and having a better or similar accuracy in predicting neural activity and behavior.Significance.Overall, multiscale SID is an accurate learning method that is particularly beneficial when efficient learning is of interest, such as for online adaptive BMIs to track non-stationary dynamics or for reducing offline training time in neuroscience investigations.

Latent Representation Learning for Multimodal Brain Activity Translation

Deep Representation Learning For Multimodal Brain Networks

Learning Active Multimodal Subspaces in the Brain

Latent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration

DSAM: A Deep Learning Framework for Analyzing Temporal and Spatial Dynamics in Brain Networks

Spatio-temporally separable non-linear latent factor learning: an application to somatomotor cortex fMRI data

SCDM: Unified Representation Learning for EEG-to-fNIRS Cross-Modal Generation in MI-BCIs

BrainLM: A foundation model for brain activity recordings

Neural Latent Aligner: Cross-trial Alignment for Learning Representations of Complex, Naturalistic Neural Data

UMBRAE: Unified Multimodal Brain Decoding

Revealing Vision-Language Integration in the Brain with Multimodal Networks

MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model

Latent Variable Algorithms for Multimodal Learning and Sensor Fusion

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data

Nonlinear latent representations of high-dimensional task-fMRI data: Unveiling cognitive and behavioral insights in heterogeneous spatial maps

Multi-Task Collaborative Pre-Training and Adaptive Token Selection: A Unified Framework for Brain Representation Learning

See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI

Joint fMRI Decoding and Encoding with Latent Embedding Alignment

Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity

A Hybrid Spatiotemporal Deep Belief Network and Sparse Representation-Based Framework Reveals Multilevel Core Functional Components in Decoding Multitask Fmri Signals.