Combination of Augmented Reality Based Brain-Computer Interface and EyeTracking for Control of a Multi-Robot System

Xiaoying Xu,Huayu Zhang,Hong Zeng,Aiguo Song,Yifei Xing,Jingtian Zhang

DOI: https://doi.org/10.1145/3674746.3674760

2024-01-01

Abstract:Even as autonomous multi-robot systems (MRS) advance rapidly, current unmanned clusters struggle to complete many real-world robotic tasks relying solely on autonomous intelligence. Therefore, the human involvement is urgently required for MRS to effectively accomplish their missions in practical applications, and the hands-on control methods have been frequently employed for the operators to input MRS control commands. Whereas, in more complex environment where the tasks require both of the operator’s hands to be engaged, the hands-on approach is insufficient to achieve dependable human-MRS interaction. Therefore, this study represents an initial exploration into a hands-free control interface utilizing augmented reality (AR) and brain-computer interface (BCI) known as an AR-based FRP-BCI, an extension of the hands-on control approach. Specifically, the operator fixate on the target command menu displayed by Microsoft HoloLens2, the control command selection function is achieved with a more natural AR-based FRP-BCI via recognizing EEG signals. During the online experiment of the AR-based FRP-BCI command selection task with the MRS, the success rate of recognization is 90.67 ± 6.80 %. The proposed AR-based FRP-BCI system enables the users to select intention targets more natural in specific real-world environments. The results highlight the capability of the hands-free AR-based FRP-BCI to enhance traditional manual MRS input devices and the establishment of a more natural interface.