Decoding imaginary handwriting trajectories with shape and time distortion loss for brain-to-text communication

Guangxiang Xu,Zebin Wang,Kedi Xu,Junming Zhu,Jianmin Zhang,Yueming Wang,Yaoyao Hao

DOI: https://doi.org/10.1101/2024.07.02.24309802

2024-11-25

Abstract:The potential to decode handwriting trajectories from brain signals has yet to be fully explored in clinical brain-computer interfaces (BCIs). One of challenges remains that the clinical BCIs mostly rely on imaginary movement due to motor deficit of the subject, which often leads to misalignment with neural activity and impedes accurate decoding. Here, we recorded intracortical neural signals from a paralyzed patient during imaginary handwriting of Chinese characters, from which the trajectories of handwriting were reconstructed and translated into texts using machine learning approach. We introduced an innovated decoding framework that incorporates a novel loss function, DILATE, to accommodate both shape and temporal distortions between movement and neural activity to account for the misalignment issue. Our method reconstructed closely resembled and human-recognizable handwriting trajectories, outperforming the conventional mean square error loss by 10% of recognition rate. Moreover, the new decoding framework enabled effective multi-day data fusion, resulting in further 15% enhancement. With a dynamic time warping approach, the recognition rate achieved up to 91.1% within a 1000-character database. Additionally, we applied our method to a previous subject who imaged handwriting of English letters, showcasing its capability for single-trail trajectory reconstruction and 13.5% higher recognition outcomes. Altogether, these findings demonstrated a new decoding scheme for BCIs that could accurately reconstruct the imaginary handwriting trajectory. This advancement paves the way for a universal brain-to-text communication system that is applicable to any written language, marking a significant leap forward in the field of neural decoding and BCI technology.