Abstract:Objective. Brain-computer interfaces aim to build an efficient communication with the world using neural signals, which may bring great benefits to human society, especially to people with physical impairments. To date, the ability to translate brain signals to effective communication outcome remains low. This work explores whether the handwriting process could serve as a potential interface with high performance. To this end, we first examined how much the scalp-recorded brain signals encode information related to handwriting and whether it is feasible to precisely retrieve the handwritten content solely from the scalp-recorded electrical data. Approach. Five participants were instructed to write the sentence 'HELLO, WORLD!' repeatedly on a tablet while their brain signals were simultaneously recorded by electroencephalography (EEG). The EEG signals were first decomposed by independent component analysis for extracting features to be used to train a convolutional neural network (CNN) to recognize the written symbols. Main results. The accuracy of the CNN-based classifier trained and applied on the same participant (training and test data separated) ranged from 76.8% to 97.0%. The accuracy of cross-participant application was more diverse, ranging from 14.7% to 58.7%. These results showed the possibility of recognizing the handwritten content directly from the scalp level brain signal. A demonstration of the recognition system in an online mode was presented. The major factor that grounded the recognition was the close association between the rich dynamics of electroencephalogram source activities and the kinematic information during the handwriting movements. Significance. This work revealed an explicit and precise mapping between scalp-level electrophysiological signals and linguistic information conveyed by handwriting, which provided a novel approach to developing brain computer interfaces that focus on semantic communication.

Handwritten character classification from EEG through continuous kinematic decoding

Towards EEG-to-Text: Handwritten Character Classification via Continuous Kinematic Decoding

Fast and Accurate Decoding of Finger Movements from ECoG Through Riemannian Features and Modern Machine Learning Techniques

Towards Scalable Handwriting Communication via EEG Decoding and Latent Embedding Integration

Enhanced Classification of Individual Finger Movements with ECoG.

Online recognition of handwritten characters from scalp-recorded brain activities during handwriting

Temporally resolved neural dynamics underlying handwriting

Rapid Decoding of Hand Gestures in Electrocorticography Using Recurrent Neural Networks.

Tracking Children’s Handwriting Learning Process Using EEG: A System Development and Validation

Motion-Based Handwriting Recognition

NeuroAiR: Deep Learning Framework for Airwriting Recognition from Scalp-recorded Neural Signals

Reconstructing Multi-Stroke Characters from Brain Signals Toward Generalizable Handwriting Brain-Computer Interfaces

Decoding Hand Movement Types and Kinematic Information From Electroencephalogram

Decoding hand movements from human EEG to control a robotic arm in a simulation environment

EEG Cortical Source Feature based Hand Kinematics Decoding using Residual CNN-LSTM Neural Network

Chrono-EEG dynamics influencing hand gesture decoding: a 10-hour study

Decoding imaginary handwriting trajectories with shape and time distortion loss for brain-to-text communication

Continuous and discrete decoding of overt speech with scalp electroencephalography (EEG)

Deep learning-based classification of fine hand movements from low frequency EEG

PreMovNet: Pre-Movement EEG-based Hand Kinematics Estimation for Grasp and Lift task

Decoding English Alphabet Letters Using EEG Phase Information.