Abstract:Characterizing the relationship between neural population activity and behavioral data is a central goal of neuroscience. While latent variable models (LVMs) are successful in describing high-dimensional time-series data, they are typically only designed for a single type of data, making it difficult to identify structure shared across different experimental data modalities. Here, we address this shortcoming by proposing an unsupervised LVM which extracts temporally evolving shared and independent latents for distinct, simultaneously recorded experimental modalities. We do this by combining Gaussian Process Factor Analysis (GPFA), an interpretable LVM for neural spiking data with temporally smooth latent space, with Gaussian Process Variational Autoencoders (GP-VAEs), which similarly use a GP prior to characterize correlations in a latent space, but admit rich expressivity due to a deep neural network mapping to observations. We achieve interpretability in our model by partitioning latent variability into components that are either shared between or independent to each modality. We parameterize the latents of our model in the Fourier domain, and show improved latent identification using this approach over standard GP-VAE methods. We validate our model on simulated multi-modal data consisting of Poisson spike counts and MNIST images that scale and rotate smoothly over time. We show that the multi-modal GP-VAE (MM-GPVAE) is able to not only identify the shared and independent latent structure across modalities accurately, but provides good reconstructions of both images and neural rates on held-out trials. Finally, we demonstrate our framework on two real world multi-modal experimental settings: Drosophila whole-brain calcium imaging alongside tracked limb positions, and Manduca sexta spike train measurements from ten wing muscles as the animal tracks a visual stimulus.

CAVACHON: a hierarchical variational autoencoder to integrate multi-modal single-cell data

Integrating single-cell multimodal epigenomic data using 1D-convolutional neural networks

Efficient and scalable integration of single-cell data using domain-adversarial and variational approximation

A multi-view latent variable model reveals cellular heterogeneity in complex tissues for paired multimodal single-cell data

Using Multi-Encoder Semi-Implicit Graph Variational Autoencoder to Analyze Single-Cell RNA Sequencing Data

HiCVAE: A Variational Auto-Encoder Framework for Simulating Hi-C Data.

Comprehensive View Embedding Learning for Single-Cell Multimodal Integration

A Versatile and Scalable Single-Cell Data Integration Algorithm Based on Domain-Adversarial and Variational Approximation.

Deep cross-omics cycle attention model for joint analysis of single-cell multi-omics data

Single-cell mosaic integration and cell state transfer with auto-scaling self-attention mechanism

Multi-view Factorization AutoEncoder with Network Constraints for Multi-omic Integrative Analysis

Simultaneous dimensionality reduction and integration for single-cell ATAC-seq data using deep learning

Autoencoders with shared and specific embeddings for multi-omics data integration

Integrated analysis of multimodal single-cell data with structural similarity

Con-AAE: contrastive cycle adversarial autoencoders for single-cell multi-omics alignment and integration

Multimodal hierarchical Variational AutoEncoders with Factor Analysis latent space

Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data

Similarity-assisted variational autoencoder for nonlinear dimension reduction with application to single-cell RNA sequencing data

Sparsely Connected Autoencoders: A Multi-Purpose Tool for Single Cell omics Analysis

InVA: Integrative Variational Autoencoder for Harmonization of Multi-modal Neuroimaging Data

The covariance environment defines cellular niches for spatial inference