Abstract:Characterizing the relationship between neural population activity and behavioral data is a central goal of neuroscience. While latent variable models (LVMs) are successful in describing high-dimensional time-series data, they are typically only designed for a single type of data, making it difficult to identify structure shared across different experimental data modalities. Here, we address this shortcoming by proposing an unsupervised LVM which extracts temporally evolving shared and independent latents for distinct, simultaneously recorded experimental modalities. We do this by combining Gaussian Process Factor Analysis (GPFA), an interpretable LVM for neural spiking data with temporally smooth latent space, with Gaussian Process Variational Autoencoders (GP-VAEs), which similarly use a GP prior to characterize correlations in a latent space, but admit rich expressivity due to a deep neural network mapping to observations. We achieve interpretability in our model by partitioning latent variability into components that are either shared between or independent to each modality. We parameterize the latents of our model in the Fourier domain, and show improved latent identification using this approach over standard GP-VAE methods. We validate our model on simulated multi-modal data consisting of Poisson spike counts and MNIST images that scale and rotate smoothly over time. We show that the multi-modal GP-VAE (MM-GPVAE) is able to not only identify the shared and independent latent structure across modalities accurately, but provides good reconstructions of both images and neural rates on held-out trials. Finally, we demonstrate our framework on two real world multi-modal experimental settings: Drosophila whole-brain calcium imaging alongside tracked limb positions, and Manduca sexta spike train measurements from ten wing muscles as the animal tracks a visual stimulus.

Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders

Multimodal normative modeling in Alzheimer Disease with introspective variational autoencoders

Normative Modeling using Multimodal Variational Autoencoders to Identify Abnormal Brain Structural Patterns in Alzheimer Disease

Multi-modal Variational Autoencoders for normative modelling across multiple imaging modalities

Normative Modeling via Conditional Variational Autoencoder and Adversarial Learning to Identify Brain Dysfunction in Alzheimer's Disease

MC-RVAE: Multi-channel recurrent variational autoencoder for multimodal Alzheimer's disease progression modelling

Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives

Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Experts

Normative Modeling for AD Diagnosis and Biomarker Identification

Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data

Analyzing heterogeneity in Alzheimer Disease using multimodal normative modeling on imaging-based ATN biomarkers

Fusing multimodal neuroimaging data with a variational autoencoder

Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis

A new VAE-GAN model to synthesize arterial spin labeling images from structural MRI

Bridging Imaging and Clinical Scores in Parkinson's Progression via Multimodal Self-Supervised Deep Learning

Bimodal PET/MRI generative reconstruction based on VAE architectures

Uncovering Neural Substrates Across Alzheimer’s Disease Stages Using Contrastive Variational Autoencoder

Incomplete multi-modal representation learning for Alzheimer's disease diagnosis

Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE

Multi-Channel Stochastic Variational Inference for the Joint Analysis of Heterogeneous Biomedical Data in Alzheimer's Disease

Expert2Coder: Capturing Divergent Brain Regions Using Mixture of Regression Experts