Abstract:The integration of deep learning and neuroscience has been advancing rapidly, which has led to improvements in the analysis of brain activity and the understanding of deep learning models from a neuroscientific perspective. The reconstruction of visual experience from human brain activity is an area that has particularly benefited: the use of deep learning models trained on large amounts of natural images has greatly improved its quality, and approaches that combine the diverse information contained in visual experiences have proliferated rapidly in recent years. In this technical paper, by taking advantage of the simple and generic framework that we proposed (Takagi and Nishimoto, CVPR 2023), we examine the extent to which various additional decoding techniques affect the performance of visual experience reconstruction. Specifically, we combined our earlier work with the following three techniques: using decoded text from brain activity, nonlinear optimization for structural image reconstruction, and using decoded depth information from brain activity. We confirmed that these techniques contributed to improving accuracy over the baseline. We also discuss what researchers should consider when performing visual reconstruction using deep generative models trained on large datasets. Please check our webpage at <a class="link-external link-https" href="https://sites.google.com/view/stablediffusion-with-brain/" rel="external noopener nofollow">this https URL</a>. Code is also available at <a class="link-external link-https" href="https://github.com/yu-takagi/StableDiffusionReconstruction" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper primarily explores how to improve brain activity-based visual image reconstruction methods by combining multiple decoding techniques. The authors utilized a previously proposed simple and general framework and added three additional techniques: 1. **Using decoded captions from brain activity**: The authors employed an image captioning model (BLIP) to predict textual descriptions of images from brain activity, and then input these descriptions into the Stable Diffusion model to generate images. 2. **Combining nonlinear optimization with Generative Adversarial Networks (GANs)**: This method first decodes information from brain activity using a GAN algorithm, and then inputs the decoded results into an image encoder module to generate images. 3. **Utilizing depth information decoded from brain activity**: To further enrich the information sources for image reconstruction, the researchers also attempted to estimate depth information separately and used it along with semantic information for conditional image generation. By integrating these techniques, the research team evaluated their impact on the performance of visual experience reconstruction. The results showed that these techniques generally improved accuracy over the baseline level, but the performance varied among different subjects, and not all metrics showed consistent improvement. Additionally, the authors conducted control analyses to investigate whether there was any overlap between the dataset used to train the Stable Diffusion model and the images in the functional magnetic resonance imaging (fMRI) data, and whether this potential "data leakage" would affect reconstruction performance. The experimental results indicated that even if there was some data overlap, it did not significantly affect the validity of the research conclusions. In summary, this paper aims to explore how to improve the quality of brain activity-based visual image reconstruction by combining multiple decoding techniques and provides quantitative evaluations to support its findings.

Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Visual Image Decoding of Brain Activities using a Dual Attention Hierarchical Latent Generative Network with Multi-Scale Feature Fusion

Natural scene reconstruction from fMRI signals using generative latent diffusion

Retrieving and reconstructing conceptually similar images from fMRI with latent diffusion models and a neuro-inspired brain decoding model

Decoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion

Deep image reconstruction from human brain activity

MindLDM: Reconstruct Visual Stimuli from Fmri Using Latent Diffusion Model

MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion

Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models

Mind-bridge: Reconstructing Visual Images Based on Diffusion Model from Human Brain Activity

Decoding human brain activity with deep learning

Reconstructing seen images from human brain activity via guided stochastic search

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Brain-optimized inference improves reconstructions of fMRI brain activity

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Visual Image Reconstructed Without Semantics from Human Brain Activity Using Linear Image Decoders and Nonlinear Noise Suppression

The brain-inspired decoder for natural visual image reconstruction

Functional diversity of visual cortex improves constraint-free natural image reconstruction from human brain activity