Abstract:Background: Image reconstruction is a critical task in brain decoding research, primarily utilizing functional magnetic resonance imaging (fMRI) data. However, due to challenges such as limited samples in fMRI data, the quality of reconstruction results often remains poor. New method: We proposed a three-stage multi-level deep fusion model (TS-ML-DFM). The model employed a three-stage training process, encompassing components such as image encoders, generators, discriminators, and fMRI encoders. In this method, we incorporated distinct supplementary features derived separately from depth images and original images. Additionally, the method integrated several components, including a random shift module, dual attention module, and multi-level feature fusion module. Results: In both qualitative and quantitative comparisons on the Horikawa17 and VanGerven10 datasets, our method exhibited excellent performance. Comparison with existing methods: For example, on the primary Horikawa17 dataset, our method was compared with other leading methods based on metrics the average hash value, histogram similarity, mutual information, structural similarity accuracy, AlexNet(2), AlexNet(5), and pairwise human perceptual similarity accuracy. Compared to the second-ranked results in each metric, the proposed method achieved improvements of 0.99%, 3.62%, 3.73%, 2.45%, 3.51%, 0.62%, and 1.03%, respectively. In terms of the SwAV top-level semantic metric, a substantial improvement of 10.53% was achieved compared to the second-ranked result in the pixel-level reconstruction methods. Conclusions: The TS-ML-DFM method proposed in this study, when applied to decoding brain visual patterns using fMRI data, has outperformed previous algorithms, thereby facilitating further advancements in research within this field.

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

Deep Learning for Under-Sampled Non-Cartesian ASL MRI Reconstruction

Natural scene reconstruction from fMRI signals using generative latent diffusion

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Reconstructing Natural Images from Human Fmri by Alternating Encoding and Decoding with Shared Autoencoder Regularization

A Survey: Challenges and Future Research Directions of fMRI Based Brain Activity Decoding Model

Deep image reconstruction from human brain activity

fMRI-based Decoding of Visual Information from Human Brain Activity: A Brief Review

More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activity

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

Deep Magnetic Resonance Image Reconstruction: Inverse Problems Meet Neural Networks

Deep MRI Reconstruction: Unrolled Optimization Algorithms Meet Neural Networks

Pre-training Deep Neural Network for Decoding Multiple Brain Regions Can Enhance Image Reconstruction

Deep-Learning Methods for Parallel Magnetic Resonance Imaging Reconstruction: A Survey of the Current Approaches, Trends, and Issues

Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning

Using Deep Learning for Visual Decoding and Reconstruction from Brain Activity: A Review

Reconstruction of natural images from evoked brain activity with a dictionary-based invertible encoding procedure

Beyond the Conventional Structural MRI: Clinical Application of Deep Learning Image Reconstruction and Synthetic MRI of the Brain

Visual Image Reconstruction from fMRI Activation Using Multi-scale Support Vector Machine Decoders.

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals