Abstract:Characterizing the relationship between neural population activity and behavioral data is a central goal of neuroscience. While latent variable models (LVMs) are successful in describing high-dimensional time-series data, they are typically only designed for a single type of data, making it difficult to identify structure shared across different experimental data modalities. Here, we address this shortcoming by proposing an unsupervised LVM which extracts temporally evolving shared and independent latents for distinct, simultaneously recorded experimental modalities. We do this by combining Gaussian Process Factor Analysis (GPFA), an interpretable LVM for neural spiking data with temporally smooth latent space, with Gaussian Process Variational Autoencoders (GP-VAEs), which similarly use a GP prior to characterize correlations in a latent space, but admit rich expressivity due to a deep neural network mapping to observations. We achieve interpretability in our model by partitioning latent variability into components that are either shared between or independent to each modality. We parameterize the latents of our model in the Fourier domain, and show improved latent identification using this approach over standard GP-VAE methods. We validate our model on simulated multi-modal data consisting of Poisson spike counts and MNIST images that scale and rotate smoothly over time. We show that the multi-modal GP-VAE (MM-GPVAE) is able to not only identify the shared and independent latent structure across modalities accurately, but provides good reconstructions of both images and neural rates on held-out trials. Finally, we demonstrate our framework on two real world multi-modal experimental settings: Drosophila whole-brain calcium imaging alongside tracked limb positions, and Manduca sexta spike train measurements from ten wing muscles as the animal tracks a visual stimulus.

InVA: Integrative Variational Autoencoder for Harmonization of Multi-modal Neuroimaging Data

Fusing multimodal neuroimaging data with a variational autoencoder

Bimodal PET/MRI generative reconstruction based on VAE architectures

Bayesian Inference for Non-Linear Forward Model by Using a VAE-Based Neural Network Structure

Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis

A new VAE-GAN model to synthesize arterial spin labeling images from structural MRI

Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex

Multi-modal Variational Autoencoders for normative modelling across multiple imaging modalities

Motion-Invariant Variational Auto-Encoding of Brain Structural Connectomes

Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE

Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations

MoVAE: Multi-Omics Variational Auto-Encoder for Cancer Subtype Detection

Compressed Sensing MRI Reconstruction Regularized by VAEs with Structured Image Covariance

Deep Variational Autoencoder for Mapping Functional Brain Networks

Multimodal Disentangled Variational Autoencoder With Game Theoretic Interpretability for Glioma Grading

Multi-source Feature Learning for Joint Analysis of Incomplete Multiple Heterogeneous Neuroimaging Data

Multi-Source Neural Variational Inference

Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data

Cross-Modal Information Recovery and Enhancement Using Multiple-Input–Multiple-Output Variational Autoencoder

Sparse-Coding Variational Auto-Encoders