Abstract:Decoding brain activity related to specific tasks, such as imagining something, is important for brain computer interface (BCI) control. While decoding of brain signals, such as functional magnetic resonance imaging (fMRI) signals and electroencephalography (EEG) signals, during observing visual images and while imagining images has been previously reported, further development of methods for improving training, performance, and interpretation of brain data was the goal of this study. We applied a Sinc-EEGNet to decode brain activity during perception and imagination of visual stimuli, and added an attention module to extract the importance of each electrode or frequency band. We also reconstructed images from brain activity by using a generative adversarial network (GAN). By combining the EEG recorded during a visual task (perception) and an imagination task, we have successfully boosted the accuracy of classifying EEG data in the imagination task and improved the quality of reconstruction by GAN. Our result indicates that the brain activity evoked during the visual task is present in the imagination task and can be used for better classification of the imagined image. By using the attention module, we can derive the spatial weights in each frequency band and contrast spatial or frequency importance between tasks from our model. Imagination tasks are classified by low frequency EEG signals over temporal cortex, while perception tasks are classified by high frequency EEG signals over occipital and frontal cortex. Combining data sets in training results in a balanced model improving classification of the imagination task without significantly changing performance in the visual task. Our approach not only improves performance and interpretability but also potentially reduces the burden on training since we can improve the accuracy of classifying a relatively hard task with high variability (imagination) by combining with the data of the relatively easy task, observing visual images.

Visual Saliency and Image Reconstruction from EEG Signals via an Effective Geometric Deep Network-Based Generative Adversarial Network

NeuroGAN: image reconstruction from EEG signals via an attention-based GAN

Salient Arithmetic Data Extraction from Brain Activity via an Improved Deep Network

EEG2IMAGE: Image Reconstruction from EEG Brain Signals

Neurocognition-inspired Design with Machine Learning

Semantics-Guided Hierarchical Feature Encoding Generative Adversarial Network for Visual Image Reconstruction From Brain Activity

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

MindCeive: Perceiving human imagination using CNN-GRU and GANs

Generating Visual Stimuli from EEG Recordings using Transformer-encoder based EEG encoder and GAN

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

Reconstructing Visual Stimulus Images from EEG Signals Based on Deep Visual Representation Model

A Visual Characteristic Land-Scape Design for EEG Signal Based on LSTM-GAN

Generative Adversarial Network (GAN) for Simulating Electroencephalography

Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

A GAN model encoded by CapsEEGNet for visual EEG encoding and image reproduction

Improving classification and reconstruction of imagined images from EEG signals

BigGAN-based Bayesian reconstruction of natural images from human brain activity

Visual Image Decoding of Brain Activities using a Dual Attention Hierarchical Latent Generative Network with Multi-Scale Feature Fusion

Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning

Deep image reconstruction from human brain activity