Abstract:Brain decoding is a field of computational neuroscience that aims to infer mental states or internal representations of perceptual inputs from measurable brain activity. In this study, we propose a novel approach to brain decoding that relies on semantic and contextual similarity. We use several fMRI datasets of natural images as stimuli and create a deep learning decoding pipeline inspired by the bottom-up and top-down processes in human vision. Our pipeline includes a linear brain-to-feature model that maps fMRI activity to semantic visual stimuli features. We assume that the brain projects visual information onto a space that is homeomorphic to the latent space represented the last layer of a pretrained neural network, which summarizes and highlights similarities and differences between concepts. These features are categorized in the latent space using a nearest-neighbor strategy, and the results are used to retrieve images or condition a generative latent diffusion model to create novel images. We demonstrate semantic classification and image retrieval on three different fMRI datasets, GOD (vision perception and imagination), BOLD5000 and NSD. In all cases a simple mapping between fMRI and a deep semantic representation of the visual stimulus resulted in meaningful classification and retrieved or generated images. We assessed quality using quantitative metrics and a human evaluation experiment that reproduces the multiplicity of conscious and unconscious criteria that humans use to evaluate image similarity. Our method achieved correct evaluation in over 80% of the test set. In summary, our study proposes a novel approach to brain decoding that relies on semantic and contextual similarity. Our results demonstrate that measurable neural correlates can be linearly mapped onto the latent space of a neural network to synthesize images that match the original content. The findings have implications for both cognitive neuroscience and artificial intelligence.

Learning visual stimulus-evoked EEG manifold for neural image classification

EEG classification for visual brain decoding with spatio-temporal and transformer based paradigms

Decoding Digital Visual Stimulation From Neural Manifold With Fuzzy Leaning on Cortical Oscillatory Dynamics

Decoding Brain Signals from Rapid-Event EEG for Visual Analysis Using Deep Learning

Deep Learning Human Mind for Automated Visual Classification

Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models

EEG Signal Classification Using Manifold Learning and Matrix-Variate Gaussian Model

Manifold Attention-Enhanced Multi-Domain Convolutional Network for Decoding Motor Imagery Intention

Decoding Visual Recognition of Objects from EEG Signals based on Attention-Driven Convolutional Neural Network

Brain-guided manifold transferring to improve the performance of spiking neural networks in image classification

Decoding human brain activity with deep learning

Human Recognition Using Deep Neural Networks and Spatial Patterns of SSVEP Signals

An intertwined neural network model for EEG classification in brain-computer interfaces

A Visual Characteristic Land-Scape Design for EEG Signal Based on LSTM-GAN

EEG classification based on Grassmann manifold and matrix recovery

Automated Classification of Cognitive Visual Objects Using Multivariate Swarm Sparse Decomposition From Multichannel EEG-MEG Signals

Eeg Based Visual Classification with Multi-Feature Joint Learning

EEG-based Image Feature Extraction for Visual Classification using Deep Learning

Retrieving and reconstructing conceptually similar images from fMRI with latent diffusion models and a neuro-inspired brain decoding model

Group-level Brain Decoding with Deep Learning

Enhancing EEG-Based MI-BCIs with Class-Specific and Subject-Specific Features Detected by Neural Manifold Analysis