Abstract:A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z - and w -latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w -latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding. Neural coding seeks to understand how the brain represents the world by modeling the relationship between stimuli and internal neural representations thereof. This field focuses on predicting brain responses to stimuli (neural encoding) and deciphering information about stimuli from brain activity (neural decoding). Recent advances in generative adversarial networks (GANs; a type of machine learning model) have enabled the creation of photorealistic images. Like the brain, GANs also have internal representations of the images they create, referred to as "latents". More recently, a new type of feature-disentangled " w -latent" of GANs has been developed that more effectively separates different image features (e.g., color; shape; texture). In our study, we presented such GAN-generated pictures to a macaque with cortical implants and found that the underlying w -latents were accurate predictors of high-level brain activity. We then used these w -latents to reconstruct the perceived images with high fidelity. The remarkable similarities between our predictions and the actual targets indicate alignment in how w -latents and neural representations represent the same stimulus, even though GANs have never been optimized on neural data. This implies a general principle of shared encoding of visual phenomena, emphasizing the importance of feature disentanglement in deeper visual areas.

Generative Modelling of Cortical Receptor Distributions from Cytoarchitectonic Images in the Macaque Brain

NeuRegenerate: A Framework for Visualizing Neurodegeneration

Probabilistic Brain Extraction in MR Images via Conditional Generative Adversarial Networks

Gradients of neurotransmitter receptor expression in the macaque cortex

Topographic variation in neurotransmitter receptor densities explains differences in intracranial EEG spectra

Metadata-conditioned generative models to synthesize anatomically-plausible 3D brain MRIs

The cytoarchitectonic landscape revealed by deep learning method facilitated precise positioning in mouse neocortex

Generative AI unlocks PET insights: brain amyloid dynamics and quantification

A complementary approach for neocortical cytoarchitecture inspection with cellular resolution imaging at whole brain scale

Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models

An Organism Starts with a Single Pix-Cell: A Neural Cellular Diffusion for High-Resolution Image Synthesis

Realistic morphology-preserving generative modelling of the brain

Solving the problem and quantifying geometric variation in neuroanatomy using generative diffeomorphic mapping

In vivo neuropil density from anatomical MRI and machine learning

Evaluation of Automated Segmentation Algorithms for Neurons in Macaque Cerebral Microscopic Images

Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain

Macaque Neuron Instance Segmentation Only with Point Annotations Based on Multiscale Fully Convolutional Regression Neural Network

Gray Matters: An Efficient Vision Transformer GAN Framework for Predicting Functional Network Connectivity Biomarkers from Brain Structure

InSpectro-Gadget: A Tool for Estimating Neurotransmitter and Neuromodulator Receptor Distributions for MRS Voxels

NeuroGen: Activation optimized image synthesis for discovery neuroscience