Abstract:Despite decades of research, much is still unknown about the computations carried out in the human face processing network. Recently, deep networks have been proposed as a computational account of human visual processing, but while they provide a good match to neural data throughout visual cortex, they lack interpretability. We introduce a method for interpreting brain activity using a new class of deep generative models, disentangled representation learning models, which learn a low-dimensional latent space that "disentangles" different semantically meaningful dimensions of faces, such as rotation, lighting, or hairstyle, in an unsupervised manner by enforcing statistical independence between dimensions. We find that the majority of our model's learned latent dimensions are interpretable by human raters. Further, these latent dimensions serve as a good encoding model for human fMRI data. We next investigate the representation of different latent dimensions across face-selective voxels. We find that low- and high-level face features are represented in posterior and anterior face-selective regions, respectively, corroborating prior models of human face recognition. Interestingly, though, we find identity-relevant and irrelevant face features across the face processing network. Finally, we provide new insight into the few "entangled" (uninterpretable) dimensions in our model by showing that they match responses in the ventral stream and carry information about facial identity. Disentangled face encoding models provide an exciting alternative to standard "black box" deep learning approaches for modeling and interpreting human brain data. We use a class of interpretable deep neural network models, disentangled variational autoencoders (dVAEs), to analyze human fMRI data. We find that a dVAE learns human interpretable dimensions of faces, such as lighting, expression, and hairstyle, and provides as good a match to human fMRI data as matched, non-disentangled models. Our disentangled encoding approach allows us to map different disentangled features to ROI and voxel activity. A decoding analysis confirms that the model separates identity relevant and irrelevant information and reveals that the remaining entangled dimensions contain identity-relevant information. Together these results highlight the use of disentangled models for more interpretable fMRI encoding than standard deep learning models.

Reconstructing faces from fMRI patterns using deep generative neural networks

Realistic Face Reenactment Via Self-Supervised Disentangling of Identity and Pose

Reconstructing controllable faces from brain activity with hierarchical multiview representations

Disentangled deep generative models reveal coding principles of the human face processing network

Brain-driven facial image reconstruction via StyleGAN inversion with improved identity consistency

Vec2Face-v2: Unveil Human Faces from their Blackbox Features via Attention-based Network in Face Recognition

Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex

Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning

DCNN-GAN: Reconstructing Realistic Image from fMRI

Semantics-Guided Hierarchical Feature Encoding Generative Adversarial Network for Visual Image Reconstruction From Brain Activity

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Learning a face space for experiments on human identity

Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons

BigGAN-based Bayesian reconstruction of natural images from human brain activity

Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN

Deep Face Decoder: Towards understanding the embedding space of convolutional networks through visual reconstruction of deep face templates

Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning

Deep image reconstruction from human brain activity

Learning brain representation using recurrent Wasserstein generative adversarial net

Double-Flow GAN model for the reconstruction of perceived faces from brain activities

One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space