"Resolution" of diploid-tetraploid tree frogs.

R. D. Maxson,L. Maxson

IF: 56.9

Science

Abstract:

What problem does this paper attempt to address?

Decoding auditory attention (in real time) with eeg

Edmund Lalor,Nima Mesgarani,Siddharth Rajaram,Adam O'Donovan,James Wright,Inyong Choi,Jonathan Brumberg,Nai Ding,Adrian KC Lee,Nils Peters,Sudarshan Ramenahalli,Jeffrey Pompe,Barbara Shinn-Cunningham,Malcolm Slaney,Shihab Shamma

2013-01-01

Abstract:BackgroundBoth magnetoencephalography and electrocorticography recordings have been used to decode which of two competing sources a listener is attending. However, it is not clear whether these techniques might work with Electroencephalography (EEG), particularly in a real-time system. We therefore set out to decode a listener’s attentional focus from EEG signals in real time, knowledge that could be incorporated into next-generation assistive listening devices.
Decoding Human Visual Colour EEG Information Using Machine Learning and Visual Evoked Potentials

Yijia Wu,Xinhua Zeng,Kaiqiang Feng,Donglai Wei,Liang Song

DOI: https://doi.org/10.11922/sciencedb.01421

2022-01-01

Abstract:With the rapid development of brain-computer interfaces (BCIs), human visual decoding, one of the important research directions of BCIs, has attracted a substantial amount of attention. However, most visual decoding studies have focused on graphic and image decoding. In this paper, we first demonstrate the possibility of building a new kind of task-irrelevant, simple and fast-stimulus BCI-based experimental paradigm that relies on visual evoked potentials (VEPs) during colour observation. Additionally, the features of visual colour information were found through reliable real-time decoding. We selected 9 subjects who did not have colour blindness to participate in our tests. These subjects were asked to observe red, green, and blue screens in turn with an interstimulus interval of 1 second. The machine learning results showed that the visual colour classification accuracy had a maximum of 93.73%. The latency evoked by visual colour stimuli was within the P300 range, i.e., 176.8 milliseconds for the red screen, 206.5 milliseconds for the green screen, and 225.3 milliseconds for the blue screen. The experimental results hereby show that the VEPs can be used for reliable colour real-time decoding.
A Method to Study Global Spatial Patterns Related to Sensory Perception in Scalp EEG.

Yusely Ruiz,Susan Pockett,Walter J. Freeman,Eduardo Gonzalez,Guang Li

DOI: https://doi.org/10.1016/j.jneumeth.2010.05.021

IF: 2.987

2010-01-01

Journal of Neuroscience Methods

Abstract:Prior studies of multichannel ECoG from animals showed that beta and gamma oscillations carried perceptual information in both local and global spatial patterns of amplitude modulation, when the subjects were trained to discriminate conditioned stimuli (CS). Here the hypothesis was tested that similar patterns could be found in the scalp EEG human subjects trained to discriminate simultaneous visual-auditory CS. Signals were continuously recorded from 64 equispaced scalp electrodes and band-pass filtered. The Hilbert transform gave the analytic phase, which segmented the EEG into temporal frames, and the analytic amplitude, which expressed the pattern in each frame as a feature vector. Methods applied to the ECoG were adapted to the EEG for systematic search of the beta–gamma spectrum, the time period after CS onset, and the scalp surface to locate patterns that could be classified with respect to type of CS. Spatial patterns of EEG amplitude modulation were found from all subjects that could be classified with respect to stimulus combination type significantly above chance levels. The patterns were found in the beta range (15–22 Hz) but not in the gamma range. They occurred in three short bursts following CS onset. They were non-local, occupying the entire array. Our results suggest that the scalp EEG can yield information about the timing of episodically synchronized brain activity in higher cognitive function, so that future studies in brain–computer interfacing can be better focused. Our methods may be most valuable for analyzing data from dense arrays with very high spatial and temporal sampling rates.
Decoding of the Neural Representation of the Visual RGB Color Model

Yijia Wu,Yanjing Mao,Kaiqiang Feng,Donglai Wei,Liang Song

DOI: https://doi.org/10.7717/peerj-cs.1376

2023-01-01

PeerJ Computer Science

Abstract:RGB color is a basic visual feature. Here we use machine learning and visual evoked potential (VEP) of electroencephalogram (EEG) data to investigate the decoding features of the time courses and space location that extract it, and whether they depend on a common brain cortex channel. We show that RGB color information can be decoded from EEG data and, with the task-irrelevant paradigm, features can be decoded across fast changes in VEP stimuli. These results are consistent with the theory of both event-related potential (ERP) and P300 mechanisms. The latency on time course is shorter and more temporally precise for RGB color stimuli than P300, a result that does not depend on a task-relevant paradigm, suggesting that RGB color is an updating signal that separates visual events. Meanwhile, distribution features are evident for the brain cortex of EEG signal, providing a space correlate of RGB color in classification accuracy and channel location. Finally, space decoding of RGB color depends on the channel classification accuracy and location obtained through training and testing EEG data. The result is consistent with channel power value distribution discharged by both VEP and electrophysiological stimuli mechanisms.
Visual Color Decoding Using Brain-Computer Interfaces

Yijia Wu,Xinhua Zeng,Kaiqiang Feng,Donglai Wei,Liang Song

DOI: https://doi.org/10.1109/INSAI54028.2021.00057

2021-01-01

Abstract:With the rapid development of Brain-Computer Interfaces (BCI), human visual decoding, as one of the important research directions of BCI, has aroused great attention. But most visual decoding researches focused on graphics decoding. In this paper, we investigate the possibility to build a new kind of BCI visual decoding based on visual color observation for the first time. We selected 10 subjects ...
Reconstructing Sources Location of Visual Color Cortex by the Task-Irrelevant Visual Stimuli Through Machine Learning Decoding

Yijia Wu,Yanni Zhang,Yanjing Mao,Kaiqiang Feng,Donglai Wei,Liang Song

DOI: https://doi.org/10.1016/j.heliyon.2022.e12287

IF: 3.776

2022-01-01

Heliyon

Abstract:Visual color sensing is generated by electrical discharges from endocranial neuronal sources that penetrate the skull and reach to the cerebral cortex. However, the space location of the source generated by this neural mechanism remains elusive. In this paper, we emulate the generation of visual color signal by task-irrelevant stimuli to activate brain neurons, where its consequences over the cerebral cortex is experimentally tracked. We first document the changes to brain color sensing using electroencephalography (EEG), and find that the sensing classification accuracy of primary visual cortex (V1) regions was positively correlated with the space correlation of visual evoked potential (VEP) power distribution under machine learning decoding. We then explore the decoded results to trace the brain activity neural source location of EEG inversion problem and assess its reconstructive possibility. We show that visual color EEG in V1 can reconstruct endocranial neuronal source location, through the machine learning decoding of channel location.
Decoding Digital Visual Stimulation From Neural Manifold With Fuzzy Leaning on Cortical Oscillatory Dynamics

Haitao Yu,Quanfa Zhao,Shanshan Li,Kai Li,Chen Liu,Jiang Wang

DOI: https://doi.org/10.3389/fncom.2022.852281

IF: 3.387

2022-03-11

Frontiers in Computational Neuroscience

Abstract:A crucial point in neuroscience is how to correctly decode cognitive information from brain dynamics for motion control and neural rehabilitation. However, due to the instability and high dimensions of electroencephalogram (EEG) recordings, it is difficult to directly obtain information from original data. Thus, in this work, we design visual experiments and propose a novel decoding method based on the neural manifold of cortical activity to find critical visual information. First, we studied four major frequency bands divided from EEG and found that the responses of the EEG alpha band (8–15 Hz) in the frontal and occipital lobes to visual stimuli occupy a prominent place. Besides, the essential features of EEG data in the alpha band are further mined via two manifold learning methods. We connect temporally consecutive brain states in the t distribution random adjacency embedded (t-SNE) map on the trial-by-trial level and find the brain state dynamics to form a cyclic manifold, with the different tasks forming distinct loops. Meanwhile, it is proved that the latent factors of brain activities estimated by t-SNE can be used for more accurate decoding and the stable neural manifold is found. Taking the latent factors of the manifold as independent inputs, a fuzzy system-based Takagi-Sugeno-Kang model is established and further trained to identify visual EEG signals. The combination of t-SNE and fuzzy learning can highly improve the accuracy of visual cognitive decoding to 81.98%. Moreover, by optimizing the features, it is found that the combination of the frontal lobe, the parietal lobe, and the occipital lobe is the most effective factor for visual decoding with 83.05% accuracy. This work provides a potential tool for decoding visual EEG signals with the help of low-dimensional manifold dynamics, especially contributing to the brain–computer interface (BCI) control, brain function research, and neural rehabilitation.

neurosciences,mathematical & computational biology
Decoding region-level visual functions from invasive EEG data

Xin-Ya Zhang,Hang Lin,Zeyu Deng,Markus Siegel,Earl K Miller,Gang Yan

DOI: https://doi.org/10.1101/2024.04.02.587853

2024-05-14

Abstract:Decoding vision is an ambitious task as it aims to transform scalar brain activity into dynamic images with refined shapes, colors and movements. In familiar environments, the brain may trigger activity that resembles specific pattern, thereby facilitating decoding. Can an artificial neural network (ANN) decipher such latent patterns? Here, we explore this question using invasive electroencephalography data from monkeys. By decoding multi-region brain activity, ANN effectively captures individual regions' functional roles as a consequence of minimizing visual errors. For example, ANN recognizes that regions V4 and LIP are involved in visual color and shape processing while MT predominantly handles visual motion, aligning with regional visual functions evident in the brain. ANN likely reconstructs vision by seizing hidden spike patterns, representing stimuli distinctly in a two-dimensional plane. Furthermore, during the encoding process of transforming visual stimuli into neuronal activity, optimal performance is achieved in regions closely associated with vision processing.

Neuroscience
Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Dongyang Li,Chen Wei,Shiying Li,Jiachen Zou,Quanying Liu

2024-04-05

Abstract:How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of fMRI-based visual decoding and reconstruction. However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for EEG-based visual reconstruction. In this study, we present an EEG-based visual reconstruction framework. It consists of a plug-and-play EEG encoder called the Adaptive Thinking Mapper (ATM), which is aligned with image embeddings, and a two-stage EEG guidance image generator that first transforms EEG features into image priors and then reconstructs the visual stimuli with a pre-trained image generator. Our approach allows EEG embeddings to achieve superior performance in image classification and retrieval tasks. Our two-stage image generation strategy vividly reconstructs images seen by humans. Furthermore, we analyzed the impact of signals from different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. We report that EEG-based visual decoding achieves SOTA performance, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. The code of ATM is available at

Human-Computer Interaction,Signal Processing,Neurons and Cognition
Decoding in the fourth dimension: Classification of temporal patterns and their generalization across locations

Alejandro Santos-Mayo,Faith Gilbert,Laura Ahumada,Caitlin Traiser,Hannah Engle,Christian Panitz,Mingzhou Ding,Andreas Keil

DOI: https://doi.org/10.1101/2024.06.19.599699

2024-06-22

Abstract:Neuroscience research has increasingly used decoding techniques, in which multivariate statistical methods identify patterns in neural data that allow the classification of experimental conditions or participant groups. Typically, the features used for decoding are spatial in nature, including voxel patterns and electrode locations. However, the strength of many neurophysiological recording techniques such as electroencephalography or magnetoencephalography is in their rich temporal, rather than spatial, content. The present report proposes a new decoding method that relies on the time information contained in neural time series. This information is then used in a subsequent step, generalization across location (GAL), which characterizes the relationship between sensor locations based on their ability to cross-decode. Two datasets are used to demonstrate usage of this method, referred to as time-GAL, involving (1) event-related potentials in response to affective pictures and (2) steady-state visual evoked potentials in response to aversively conditioned grating stimuli. In both cases, experimental conditions were successfully decoded based on the temporal features contained in the neural time series. Cross-decoding occurred in regions known to be involved in visual and affective processing. We conclude that the time-GAL approach holds promise for analyzing neural time series from a wide range of paradigms and measurement domains providing an assumption-free method to quantifying differences in temporal patterns of neural information processing and whether these patterns are shared across sensor locations.

Neuroscience
EEG-based Decoding of Selective Visual Attention in Superimposed Videos

Yuanyuan Yao,Wout De Swaef,Simon Geirnaert,Alexander Bertrand

2024-09-19

Abstract:Selective attention enables humans to efficiently process visual stimuli by enhancing important locations or objects and filtering out irrelevant information. Locating visual attention is a fundamental problem in neuroscience with potential applications in brain-computer interfaces. Conventional paradigms often use synthetic stimuli or static images, but visual stimuli in real life contain smooth and highly irregular dynamics. In this study, we show that these irregular dynamics in natural videos can be decoded from electroencephalography (EEG) signals to perform selective visual attention decoding. To this end, we propose an experimental paradigm in which participants attend to one of two superimposed videos, each showing a center-aligned person performing a stage act. We then train a stimulus-informed decoder to extract EEG signal components that are correlated with the motion patterns of the attended object, and show that this decoder can be used on unseen data to detect which of both objects is attended. Eye movements are also found to be correlated to the motion patterns in the attended video, despite the spatial overlap between the target and the distractor. We further show that these eye movements do not dominantly drive the EEG-based decoding and that complementary information exists in EEG and gaze data. Moreover, our results indicate that EEG also captures information about unattended objects. To our knowledge, this study is the first to explore EEG-based selective visual attention decoding on natural videos, opening new possibilities for experiment design in related fields.

Signal Processing,Neurons and Cognition
Spatial scale and distribution of neurovascular signals underlying decoding of orientation and eye of origin from fMRI data

Jonas Larsson,Charlotte Harrison,Jade Jackson,Seung-Mock Oh,Vaida Zeringyte

DOI: https://doi.org/10.1152/jn.00590.2016

2017-02-01

Abstract:Multivariate pattern analysis of functional magnetic resonance imaging (fMRI) data is widely used, yet the spatial scales and origin of neurovascular signals underlying such analyses remain unclear. We compared decoding performance for stimulus orientation and eye of origin from fMRI measurements in human visual cortex with predictions based on the columnar organization of each feature and estimated the spatial scales of patterns driving decoding. Both orientation and eye of origin could be decoded significantly above chance in early visual areas (V1-V3). Contrary to predictions based on a columnar origin of response biases, decoding performance for eye of origin in V2 and V3 was not significantly lower than that in V1, nor did decoding performance for orientation and eye of origin differ significantly. Instead, response biases for both features showed large-scale organization, evident as a radial bias for orientation, and a nasotemporal bias for eye preference. To determine whether these patterns could drive classification, we quantified the effect on classification performance of binning voxels according to visual field position. Consistent with large-scale biases driving classification, binning by polar angle yielded significantly better decoding performance for orientation than random binning in V1-V3. Similarly, binning by hemifield significantly improved decoding performance for eye of origin. Patterns of orientation and eye preference bias in V2 and V3 showed a substantial degree of spatial correlation with the corresponding patterns in V1, suggesting that response biases in these areas originate in V1. Together, these findings indicate that multivariate classification results need not reflect the underlying columnar organization of neuronal response selectivities in early visual areas.NEW & NOTEWORTHY Large-scale response biases can account for decoding of orientation and eye of origin in human early visual areas V1-V3. For eye of origin this pattern is a nasotemporal bias; for orientation it is a radial bias. Differences in decoding performance across areas and stimulus features are not well predicted by differences in columnar-scale organization of each feature. Large-scale biases in extrastriate areas are spatially correlated with those in V1, suggesting biases originate in primary visual cortex.
Brain decoding: toward real-time reconstruction of visual perception

Yohann Benchetrit,Hubert Banville,Jean-Rémi King

2024-03-14

Abstract:In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution ($\approx$0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution ($\approx$5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that high-level visual features can be decoded from MEG signals, although the same approach applied to 7T fMRI also recovers better low-level features. Overall, these results, while preliminary, provide an important step towards the decoding -- in real-time -- of the visual processes continuously unfolding within the human brain.

Image and Video Processing,Artificial Intelligence,Machine Learning,Neurons and Cognition
Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

Zhang Yi-Jun,Yu Zhao-Fei,Liu Jian. K.,Huang Tie-Jun

DOI: https://doi.org/10.1007/s11633-022-1335-2

2022-01-01

Abstract:Vision plays a peculiar role in intelligence. Visual information, forming a large part of the sensory information, is fed into the human brain to formulate various types of cognition and behaviours that make humans become intelligent agents. Recent advances have led to the development of brain-inspired algorithms and models for machine vision. One of the key components of these methods is the utilization of the computational principles underlying biological neurons. Additionally, advanced experimental neuroscience techniques have generated different types of neural signals that carry essential visual information. Thus, there is a high demand for mapping out functional models for reading out visual information from neural signals. Here, we briefly review recent progress on this issue with a focus on how machine learning techniques can help in the development of models for contending various types of neural signals, from fine-scale neural spikes and single-cell calcium imaging to coarse-scale electroencephalography (EEG) and functional magnetic resonance imaging recordings of brain signals.
Visual Neural Decoding via Improved Visual-EEG Semantic Consistency

Hongzhou Chen,Lianghua He,Yihang Liu,Longzhen Yang

2024-08-13

Abstract:Visual neural decoding refers to the process of extracting and interpreting original visual experiences from human brain activity. Recent advances in metric learning-based EEG visual decoding methods have delivered promising results and demonstrated the feasibility of decoding novel visual categories from brain activity. However, methods that directly map EEG features to the CLIP embedding space may introduce mapping bias and cause semantic inconsistency among features, thereby degrading alignment and impairing decoding performance. To further explore the semantic consistency between visual and neural signals. In this work, we construct a joint semantic space and propose a Visual-EEG Semantic Decouple Framework that explicitly extracts the semantic-related features of these two modalities to facilitate optimal alignment. Specifically, a cross-modal information decoupling module is introduced to guide the extraction of semantic-related information from modalities. Then, by quantifying the mutual information between visual image and EEG features, we observe a strong positive correlation between the decoding performance and the magnitude of mutual information. Furthermore, inspired by the mechanisms of visual object understanding from neuroscience, we propose an intra-class geometric consistency approach during the alignment process. This strategy maps visual samples within the same class to consistent neural patterns, which further enhances the robustness and the performance of EEG visual decoding. Experiments on a large Image-EEG dataset show that our method achieves state-of-the-art results in zero-shot neural decoding tasks.

Computer Vision and Pattern Recognition,Human-Computer Interaction
What are we really decoding? Unveiling biases in EEG-based decoding of the spatial focus of auditory attention

Iustina Rotaru,Simon Geirnaert,Nicolas Heintz,Iris Van de Ryck,Alexander Bertrand,Tom Francart

DOI: https://doi.org/10.1088/1741-2552/ad2214

IF: 5.043

2024-01-25

Journal of Neural Engineering

Abstract:Objective. Spatial auditory attention decoding (Sp-AAD) refers to the task of identifying the direction of the speaker to which a person is attending in a multi-talker setting, based on the listener's neural recordings, e.g., electroencephalography (EEG). The goal of this study is to thoroughly investigate potential biases when training such Sp-AAD decoders on EEG data, particularly eye-gaze biases and latent trial-dependent confounds, which may result in Sp-AAD models that decode eye-gaze or trial-specific fingerprints rather than spatial auditory attention. Approach. We designed a two-speaker audiovisual Sp-AAD protocol in which the spatial auditory and visual attention were enforced to be either congruent or incongruent, and we recorded EEG data from sixteen participants undergoing several trials recorded at distinct timepoints. We trained a simple linear model for Sp-AAD based on common spatial patterns (CSP) filters in combination with either linear discriminant analysis (LDA) or k-means clustering, and evaluated them both across- and within-trial. Main results. We found that even a simple linear Sp-AAD model is susceptible to overfitting to confounding signal patterns such as eye-gaze and trial fingerprints (e.g., due to feature shifts across trials), resulting in artificially high decoding accuracies. Furthermore, we found that changes in the EEG signal statistics across trials deteriorate the trial generalization of the classifier, even when the latter is retrained on the test trial with an unsupervised algorithm. Significance. Collectively, our findings confirm that there exist subtle biases and confounds that can strongly interfere with the decoding of spatial auditory attention from EEG. It is expected that more complicated non-linear models based on deep neural networks, which are often used for Sp-AAD, are even more vulnerable to such biases. Future work should perform experiments and model evaluations that avoid and/or control for such biases in Sp-AAD tasks.

engineering, biomedical,neurosciences
Neuro-3D: Towards 3D Visual Decoding from EEG Signals

Zhanqiang Guo,Jiamin Wu,Yonghao Song,Weijian Mai,Qihao Zheng,Wanli Ouyang,Chunfeng Song

2024-11-19

Abstract:Human's perception of the visual world is shaped by the stereo processing of 3D information. Understanding how the brain perceives and processes 3D visual stimuli in the real world has been a longstanding endeavor in neuroscience. Towards this goal, we introduce a new neuroscience task: decoding 3D visual perception from EEG signals, a neuroimaging technique that enables real-time monitoring of neural dynamics enriched with complex visual cues. To provide the essential benchmark, we first present EEG-3D, a pioneering dataset featuring multimodal analysis data and extensive EEG recordings from 12 subjects viewing 72 categories of 3D objects rendered in both videos and images. Furthermore, we propose Neuro-3D, a 3D visual decoding framework based on EEG signals. This framework adaptively integrates EEG features derived from static and dynamic stimuli to learn complementary and robust neural representations, which are subsequently utilized to recover both the shape and color of 3D objects through the proposed diffusion-based colored point cloud decoder. To the best of our knowledge, we are the first to explore EEG-based 3D visual decoding. Experiments indicate that Neuro-3D not only reconstructs colored 3D objects with high fidelity, but also learns effective neural representations that enable insightful brain region analysis. The dataset and associated code will be made publicly available.

Computer Vision and Pattern Recognition
An introduction to time-resolved decoding analysis for M/EEG

Thomas A. Carlson,Tijl Grootswagers,Amanda K. Robinson

DOI: https://doi.org/10.48550/arXiv.1905.04820

2019-05-13

Neurons and Cognition

Abstract:The human brain is constantly processing and integrating information in order to make decisions and interact with the world, for tasks from recognizing a familiar face to playing a game of tennis. These complex cognitive processes require communication between large populations of neurons. The non-invasive neuroimaging methods of electroencephalography (EEG) and magnetoencephalography (MEG) provide population measures of neural activity with millisecond precision that allow us to study the temporal dynamics of cognitive processes. However, multi-sensor M/EEG data is inherently high dimensional, making it difficult to parse important signal from noise. Multivariate pattern analysis (MVPA) or "decoding" methods offer vast potential for understanding high-dimensional M/EEG neural data. MVPA can be used to distinguish between different conditions and map the time courses of various neural processes, from basic sensory processing to high-level cognitive processes. In this chapter, we discuss the practical aspects of performing decoding analyses on M/EEG data as well as the limitations of the method, and then we discuss some applications for understanding representational dynamics in the human brain.
Decoding Brain Signals from Rapid-Event EEG for Visual Analysis Using Deep Learning

Madiha Rehman,Humaira Anwer,Helena Garay,Josep Alemany Iturriaga,Isabel De la Torre Díez,Hafeez ur Rehman Siddiqui,Saleem Ullah

DOI: https://doi.org/10.3390/s24216965

IF: 3.9

2024-10-31

Sensors

Abstract:The perception and recognition of objects around us empower environmental interaction. Harnessing the brain's signals to achieve this objective has consistently posed difficulties. Researchers are exploring whether the poor accuracy in this field is a result of the design of the temporal stimulation (block versus rapid event) or the inherent complexity of electroencephalogram (EEG) signals. Decoding perceptive signal responses in subjects has become increasingly complex due to high noise levels and the complex nature of brain activities. EEG signals have high temporal resolution and are non-stationary signals, i.e., their mean and variance vary overtime. This study aims to develop a deep learning model for the decoding of subjects' responses to rapid-event visual stimuli and highlights the major factors that contribute to low accuracy in the EEG visual classification task.The proposed multi-class, multi-channel model integrates feature fusion to handle complex, non-stationary signals. This model is applied to the largest publicly available EEG dataset for visual classification consisting of 40 object classes, with 1000 images in each class. Contemporary state-of-the-art studies in this area investigating a large number of object classes have achieved a maximum accuracy of 17.6%. In contrast, our approach, which integrates Multi-Class, Multi-Channel Feature Fusion (MCCFF), achieves a classification accuracy of 33.17% for 40 classes. These results demonstrate the potential of EEG signals in advancing EEG visual classification and offering potential for future applications in visual machine models.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Yanchen Wang,Adam Turnbull,Tiange Xiang,Yunlong Xu,Sa Zhou,Adnan Masoud,Shekoofeh Azizi,Feng Vankee Lin,Ehsan Adeli

2024-11-12

Abstract:Neural decoding, the process of understanding how brain activity corresponds to different stimuli, has been a primary objective in cognitive sciences. Over the past three decades, advancements in functional Magnetic Resonance Imaging and machine learning have greatly improved our ability to map visual stimuli to brain activity, especially in the visual cortex. Concurrently, research has expanded into decoding more complex processes like language and memory across the whole brain, utilizing techniques to handle greater variability and improve signal accuracy. We argue that "seeing" involves more than just mapping visual stimuli onto the visual cortex; it engages the entire brain, as various emotions and cognitive states can emerge from observing different scenes. In this paper, we develop algorithms to enhance our understanding of visual processes by incorporating whole-brain activation maps while individuals are exposed to visual stimuli. We utilize large-scale fMRI encoders and Image generative models pre-trained on large public datasets, which are then fine-tuned through Image-fMRI contrastive learning. Our models hence can decode visual experience across the entire cerebral cortex, surpassing the traditional confines of the visual cortex. We first compare our method with state-of-the-art approaches to decoding visual processing and show improved predictive semantic accuracy by 43%. A network ablation analysis suggests that beyond the visual cortex, the default mode network contributes most to decoding stimuli, in line with the proposed role of this network in sense-making and semantic processing. Additionally, we implemented zero-shot imagination decoding on an extra validation dataset, achieving a p-value of 0.0206 for mapping the reconstructed images and ground-truth text stimuli, which substantiates the model's capability to capture semantic meanings across various scenarios.

Computer Vision and Pattern Recognition

"Resolution" of diploid-tetraploid tree frogs.

Decoding auditory attention (in real time) with eeg

Decoding Human Visual Colour EEG Information Using Machine Learning and Visual Evoked Potentials

A Method to Study Global Spatial Patterns Related to Sensory Perception in Scalp EEG.

Decoding of the Neural Representation of the Visual RGB Color Model

Visual Color Decoding Using Brain-Computer Interfaces

Reconstructing Sources Location of Visual Color Cortex by the Task-Irrelevant Visual Stimuli Through Machine Learning Decoding

Decoding Digital Visual Stimulation From Neural Manifold With Fuzzy Leaning on Cortical Oscillatory Dynamics

Decoding region-level visual functions from invasive EEG data

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Decoding in the fourth dimension: Classification of temporal patterns and their generalization across locations

EEG-based Decoding of Selective Visual Attention in Superimposed Videos

Spatial scale and distribution of neurovascular signals underlying decoding of orientation and eye of origin from fMRI data

Brain decoding: toward real-time reconstruction of visual perception

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

Visual Neural Decoding via Improved Visual-EEG Semantic Consistency

What are we really decoding? Unveiling biases in EEG-based decoding of the spatial focus of auditory attention

Neuro-3D: Towards 3D Visual Decoding from EEG Signals

An introduction to time-resolved decoding analysis for M/EEG

Decoding Brain Signals from Rapid-Event EEG for Visual Analysis Using Deep Learning

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models