Abstract:OBJECTIVE: Covert aspects of ongoing user mental states provide key context information for user-aware human computer interactions. In this paper, we focus on the problem of estimating the vigilance of users using EEG and EOG signals.APPROACH: The PERCLOS index as vigilance annotation is obtained from eye tracking glasses. To improve the feasibility and wearability of vigilance estimation devices for real-world applications, we adopt a novel electrode placement for forehead EOG and extract various eye movement features, which contain the principal information of traditional EOG. We explore the effects of EEG from different brain areas and combine EEG and forehead EOG to leverage their complementary characteristics for vigilance estimation. Considering that the vigilance of users is a dynamic changing process because the intrinsic mental states of users involve temporal evolution, we introduce continuous conditional neural field and continuous conditional random field models to capture dynamic temporal dependency.MAIN RESULTS: We propose a multimodal approach to estimating vigilance by combining EEG and forehead EOG and incorporating the temporal dependency of vigilance into model training. The experimental results demonstrate that modality fusion can improve the performance compared with a single modality, EOG and EEG contain complementary information for vigilance estimation, and the temporal dependency-based models can enhance the performance of vigilance estimation. From the experimental results, we observe that theta and alpha frequency activities are increased, while gamma frequency activities are decreased in drowsy states in contrast to awake states.SIGNIFICANCE: The forehead setup allows for the simultaneous collection of EEG and EOG and achieves comparative performance using only four shared electrodes in comparison with the temporal and posterior sites.

A MultiModal Vigilance (MMV) dataset during RSVP and SSVEP brain-computer interface tasks

A multimodal approach to estimating vigilance in SSVEP-based BCI

Vigilance Estimating in SSVEP-Based BCI Using Multimodal Signals

Investigating EEG-based Cross-Session and Cross-Task Vigilance Estimation in BCI Systems.

A multimodal approach to estimating vigilance using EEG and forehead EOG

Steady-State Visual Evoked Potential-Based Brain–Computer Interface System for Enhanced Human Activity Monitoring and Assessment

A Benchmark Dataset for RSVP-Based Brain–Computer Interfaces

Dual-Alpha: a Large EEG Study for Dual-Frequency SSVEP Brain-Computer Interface.

Multi-frequency steady-state visual evoked potential dataset

An Open Dataset for Wearable SSVEP-Based Brain-Computer Interfaces

Characterization of SSMVEP-based EEG Signals Using Multiplex Limited Penetrable Horizontal Visibility Graph.

An EEG Dataset for Studying Asynchronous Steady-State Visual Evoked Potential (SSVEP) Based Brain Computer Interfaces

A Fine-Grained Domain Adaptation Method for Cross-Session Vigilance Estimation in SSVEP-Based BCI

Designing Experimental Protocols to Elicit Vigilance Variations in Controlled Laboratory Settings

A Benchmark Dataset for SSVEP-Based Brain–Computer Interfaces

Multimodal Vigilance Estimation with Modality-Pairwise Contrastive Loss.

Multivariate empirical mode decomposition and multiscale entropy analysis of EEG signals from SSVEP-based BCI system

An open dataset for human SSVEPs in the frequency range of 1-60 Hz

Vigilance Detection Based on Sparse Representation of EEG.

Capsule Attention for Multimodal EEG-EOG Representation Learning With Application to Driver Vigilance Estimation

A large EEG dataset for studying cross-session variability in motor imagery brain-computer interface