Decoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion

Jingyuan Sun,Mingxiao Li,Marie-Francine Moens

2023-09-30

Abstract:Reconstructing visual stimuli from human brain activities provides a promising opportunity to advance our understanding of the brain's visual system and its connection with computer vision models. Although deep generative models have been employed for this task, the challenge of generating high-quality images with accurate semantics persists due to the intricate underlying representations of brain signals and the limited availability of parallel data. In this paper, we propose a two-phase framework named Contrast and Diffuse (CnD) to decode realistic images from functional magnetic resonance imaging (fMRI) recordings. In the first phase, we acquire representations of fMRI data through self-supervised contrastive learning. In the second phase, the encoded fMRI representations condition the diffusion model to reconstruct visual stimulus through our proposed concept-aware conditioning method. Experimental results show that CnD reconstructs highly plausible images on challenging benchmarks. We also provide a quantitative interpretation of the connection between the latent diffusion model (LDM) components and the human brain's visual system. In summary, we present an effective approach for reconstructing visual stimuli based on human brain activity and offer a novel framework to understand the relationship between the diffusion model and the human brain visual system.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of decoding realistic images from brain activity. Specifically, the authors propose a two-stage framework called Contrast and Diffuse (CnD), which utilizes contrastive self-supervised learning and a latent diffusion model to decode high-quality and semantically accurate images from functional magnetic resonance imaging (fMRI) recordings.

### Problems the paper attempts to solve:
1. **Generating high-quality images**: Current methods face challenges in generating high-resolution and semantically accurate images, especially when dealing with complex brain signal representations and limited datasets.
2. **Understanding the visual system**: Reconstructing visual stimuli to further understand the human brain's visual system and explore its connections with computer vision models.
3. **Brain-computer interfaces**: Developing systems that can express thoughts and intentions through brain signals, particularly for patients with motor impairments.

### Main contributions:
1. Proposed a two-stage framework based on contrastive self-supervised learning and a latent diffusion model to decode high-quality, semantically similar images from fMRI signals.
2. Introduced a method to analyze deep generative models from a biological perspective.
3. Demonstrated how diffusion models can incorporate information from the human visual system to reconstruct realistic images.

Through the above methods, the authors not only improved the quality of image reconstruction but also delved into the relationship between the components of diffusion models and the human brain's visual system.