STREAMS: An Assistive Multimodal AI Framework for Empowering Biosignal Based Robotic Controls

Ali Rabiee,Sima Ghafoori,Xiangyu Bai,Sarah Ostadabbas,Reza Abiri

2024-10-04

Abstract:End-effector based assistive robots face persistent challenges in generating smooth and robust trajectories when controlled by human's noisy and unreliable biosignals such as muscle activities and brainwaves. The produced endpoint trajectories are often jerky and imprecise to perform complex tasks such as stable robotic grasping. We propose STREAMS (Self-Training Robotic End-to-end Adaptive Multimodal Shared autonomy) as a novel framework leveraged deep reinforcement learning to tackle this challenge in biosignal based robotic control systems. STREAMS blends environmental information and synthetic user input into a Deep Q Learning Network (DQN) pipeline for an interactive end-to-end and self-training mechanism to produce smooth trajectories for the control of end-effector based robots. The proposed framework achieved a high-performance record of 98% in simulation with dynamic target estimation and acquisition without any pre-existing datasets. As a zero-shot sim-to-real user study with five participants controlling a physical robotic arm with noisy head movements, STREAMS (as an assistive mode) demonstrated significant improvements in trajectory stabilization, user satisfaction, and task performance reported as a success rate of 83% compared to manual mode which was 44% without any task support. STREAMS seeks to improve biosignal based assistive robotic controls by offering an interactive, end-to-end solution that stabilizes end-effector trajectories, enhancing task performance and accuracy.

Robotics

What problem does this paper attempt to address?

The problem this paper attempts to address is the ongoing challenge faced by assistive robots controlled by biological signals (such as muscle activity and brain waves) in generating smooth and stable end-effector trajectories. These biological signals are often noisy and unreliable, leading to imprecise trajectories of the robot's end-effector, making it difficult to perform complex tasks such as stable grasping operations.

Specifically, the paper proposes a new framework named STREAMS (Self-Training Robotic End-to-end Adaptive Multimodal Shared autonomy) that utilizes deep reinforcement learning to tackle this issue. STREAMS achieves an interactive end-to-end self-training mechanism by integrating environmental information and synthetic user inputs into the Deep Q-Learning Network (DQN) pipeline, thereby generating smooth trajectories for controlling the robot's end-effector.

The main contributions of the paper include:
1. Proposing an adaptive trajectory generation method that can produce smooth and reliable paths even under noisy and imprecise biological signal control inputs in complex and dynamic environments.
2. Developing an interactive end-to-end multimodal framework that directly transforms unreliable biological signal inputs and environmental perception into appropriate robot actions, eliminating the need for intermediate representations or explicit intent recognition.
3. Designing a DQN-based self-training framework that eliminates the dependency on any dataset by utilizing data that simulates real-world biological signal noise.
4. Demonstrating zero-shot transfer from simulation to real-world applications and validating the framework's generality and adaptability in real-world scenarios through user studies.

Through these methods, STREAMS aims to improve assistive robot control based on biological signals, providing an interactive end-to-end solution that stabilizes the end-effector's trajectory and enhances task performance and accuracy.