Abstract:Neurorobotics is one of the most ambitious fields in robotics, driving integration of interdisciplinary data and knowledge. One of the most productive areas of interdisciplinary research in this area has been the implementation of biologically-inspired mechanisms in the development of autonomous systems. Specifically, enabling such systems to display adaptive behavior such as learning from good and bad outcomes, has been achieved by quantifying and understanding the neural mechanisms of the brain networks mediating adaptive behaviors in humans and animals. For example, associative learning from aversive or dangerous outcomes is crucial for an autonomous system, to avoid dangerous situations in the future. A body of neuroscience research has suggested that the neurocomputations in the human brain during associative learning involve re-shaping of sensory responses. The nature of these adaptive changes in sensory processing during learning however are not yet well enough understood to be readily implemented into on-board algorithms for robotics application. Toward this overall goal, we record the simultaneous electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI), characterizing one candidate mechanism, i.e., large-scale brain oscillations. The present report examines the use of Functional Source Separation (FSS) as an optimization step in EEG-fMRI fusion that harnesses timing information to constrain the solutions that satisfy physiological assumptions. We applied this approach to the voxel-wise correlation of steady-state visual evoked potential (ssVEP) amplitude and blood oxygen level-dependent imaging (BOLD), across both time series. The results showed the benefit of FSS for the extraction of robust ssVEP signals during simultaneous EEG-fMRI recordings. Applied to data from a 3-phase aversive conditioning paradigm, the correlation maps across the three phases (habituation, acquisition, extinction) show converging results, notably major overlapping areas in both primary and extended visual cortical regions, including calcarine sulcus, lingual cortex, and cuneus. In addition, during the acquisition phase when aversive learning occurs, we observed additional correlations between ssVEP and BOLD in the anterior cingulate cortex (ACC) as well as the precuneus and superior temporal gyrus.

Multisensory integration of dynamic emotional faces and voices: method for simultaneous EEG-fMRI measurements

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Integrating high-density ERP and fMRI measures of face-elicited brain activity in 9–12-year-old children: An ERP source localization study

Integration of Simultaneous Fmri and EEG Source Localization in Emotional Decision Problems

Functional Source Separation for EEG-fMRI Fusion: Application to Steady-State Visual Evoked Potentials

Cortical Surface Alignment in Multi-Subject Spatiotemporal Independent EEG Source Imaging

Integrative Interaction of Emotional Speech in Audio-Visual Modality

Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement

A study of neural mechanism in emotion regulation by simultaneous recording of EEG and fMRI based on ICA

Mapping single-trial EEG records on the cortical surface through a spatiotemporal modality

Simultaneous EEG-fMRI: Trial level spatio-temporal fusion for hierarchically reliable information discovery

Feature Fusion Analysis of Simultaneously Recorded EEG-fMRI in Emotion Cognitive Reappraisal

Augmenting interictal mapping with neurovascular coupling biomarkers by structured factorization of epileptic EEG and fMRI data

Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI

Strategic Integration: A Cross-Disciplinary Review of the fNIRS-EEG Dual-Modality Imaging System for Delivering Multimodal Neuroimaging to Applications

Accumulated Source Imaging Of Brain Activity With Both Low And High-Frequency Neuromagnetic Signals

Early soft and flexible fusion of electroencephalography and functional magnetic resonance imaging via double coupled matrix tensor factorization for multisubject group analysis

Integration of EEG/MEG with MRI and Fmri in Functional Neuroimaging

Emotion Recognition Based on a EEG–fNIRS Hybrid Brain Network in the Source Space

A Functional MRI Paradigm for Efficient Mapping of Memory Encoding Across Sensory Conditions

Dynamic functional connectivity estimation for neurofeedback emotion regulation paradigm with simultaneous EEG-fMRI analysis