Abstract:Background and objective: The brain-computer interface (BCI) system based on steady-state visual evoked potentials (SSVEP) is expected to help disabled patients achieve alternative prosthetic hand assistance. However, the existing study still has some shortcomings in interaction aspects such as stimulus paradigm and control logic. The purpose of this study is to innovate the visual stimulus paradigm and asynchronous decoding/control strategy by integrating augmented reality technology, and propose an asynchronous pattern recognition algorithm, thereby improving the interaction logic and practical application capabilities of the prosthetic hand with the BCI system. Methods: An asynchronous visual stimulus paradigm based on an augmented reality (AR) interface was proposed in this paper, in which there were 8 control modes, including Grasp, Put down, Pinch, Point, Fist, Palm push, Hold pen, and Initial. According to the attentional orienting characteristics of the paradigm, a novel asynchronous pattern recognition algorithm that combines center extended canonical correlation analysis and support vector machine (Center-ECCA-SVM) was proposed. Then, this study proposed an intelligent BCI system switch based on a deep learning object detection algorithm (YOLOv4) to improve the level of user interaction. Finally, two experiments were designed to test the performance of the brain-controlled prosthetic hand system and its practical performance in real scenarios. Results: Under the AR paradigm of this study, compared with the liquid crystal display (LCD) paradigm, the average SSVEP spectrum amplitude of multiple subjects increased by 17.41%, and the signal-noise ratio (SNR) increased by 3.52%. The average stimulus pattern recognition accuracy was 96.71 ± 3.91%, which was 2.62% higher than the LCD paradigm. Under the data analysis time of 2s, the Center-ECCA-SVM classifier obtained 94.66 ± 3.87% and 97.40 ± 2.78% asynchronous pattern recognition accuracy under the Normal metric and the Tolerant metric, respectively. And the YOLOv4-tiny model achieves a speed of 25.29fps and a 96.4% confidence in the prosthetic hand in real-time detection. Finally, the brain-controlled prosthetic hand helped the subjects to complete 4 kinds of daily life tasks in the real scene, and the time-consuming were all within an acceptable range, which verified the effectiveness and practicability of the system. Conclusion: This research is based on improving the user interaction level of the prosthetic hand with the BCI system, and has made improvements in the SSVEP paradigm, asynchronous pattern recognition, interaction, and control logic. Furthermore, it also provides support for BCI areas for alternative prosthetic control, and movement disorder rehabilitation programs.

Visual task performance in the blind with the BrainPort V100 Vision Aid

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

A Novel Hybrid BCI Web Browser Based on SSVEP and Eye-Tracking.

Brain Machine Interfaces for Vision Restoration: The Current State of Cortical Visual Prosthetics

A novel brain-controlled prosthetic hand method integrating AR-SSVEP augmentation, asynchronous control, and machine vision assistance

C-Sight Visual Prostheses For The Blind - Optic Nerve Stimulation With Penetrating Electrode Array

Functional performance of a vibrotactile sensory substitution device in people with profound vision loss

Brain Vital Signs: Expanding From the Auditory to Visual Modality

A Virtual Vision Navigation System for The Blind Using Wearable Touch-vision Devices

A Novel Head-Mounted Navigation Device Based on Vibrotactile Stimulation

[Visual object detection system based on augmented reality and steady-state visual evoked potential]

A narrative review of cortical visual prosthesis systems: the latest progress and significance of nanotechnology for the future

Can visual cortex non-invasive brain stimulation improve normal visual function? A systematic review and meta-analysis

Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device

Portable multi-focal visual evoked potential diagnostics for multiple sclerosis/optic neuritis patients

Exploration of a novel physical therapy protocol that uses a sensory substitution device to improve the standing postural balance of children with balance disorders

Human-computer interaction for augmentative communication using a visual feedback system

[Research Progress of Visual Prosthesis].

Shape Discrimination Using the Tongue: Implications for A Visual-To-Tactile Sensory Substitution Device

Brain–computer interfaces based on code-modulated visual evoked potentials (c-VEP): a literature review

Comparison of Visual Stimuli for Steady-State Visual Evoked Potential-Based Brain-Computer Interfaces in Virtual Reality Environment in terms of Classification Accuracy and Visual Comfort