Touchless interactive teaching of soft robots through flexible bimodal sensory interfaces

Wenbo Liu,Youning Duo,Jiaqi Liu,Feiyang Yuan,Lei Li,Luchen Li,Gang Wang,Bohan Chen,Siqi Wang,Hui Yang,Yuchen Liu,Yanru Mo,Yun Wang,Bin Fang,Fuchun Sun,Xilun Ding,Chi Zhang,Li Wen

DOI: https://doi.org/10.1038/s41467-022-32702-5

IF: 16.6

2022-08-26

Nature Communications

Abstract:Abstract In this paper, we propose a multimodal flexible sensory interface for interactively teaching soft robots to perform skilled locomotion using bare human hands. First, we develop a flexible bimodal smart skin (FBSS) based on triboelectric nanogenerator and liquid metal sensing that can perform simultaneous tactile and touchless sensing and distinguish these two modes in real time. With the FBSS, soft robots can react on their own to tactile and touchless stimuli. We then propose a distance control method that enabled humans to teach soft robots movements via bare hand-eye coordination. The results showed that participants can effectively teach a self-reacting soft continuum manipulator complex motions in three-dimensional space through a “shifting sensors and teaching” method within just a few minutes. The soft manipulator can repeat the human-taught motions and replay them at different speeds. Finally, we demonstrate that humans can easily teach the soft manipulator to complete specific tasks such as completing a pen-and-paper maze, taking a throat swab, and crossing a barrier to grasp an object. We envision that this user-friendly, non-programmable teaching method based on flexible multimodal sensory interfaces could broadly expand the domains in which humans interact with and utilize soft robots.

multidisciplinary sciences

What problem does this paper attempt to address?

The paper attempts to address the problem of how to efficiently and flexibly "teach" soft robots to perform specific tasks through human-computer interaction. Specifically, the paper focuses on two challenges:

1. **Multimodal, multifunctional, and robust flexible sensing devices**: To achieve effective interaction between humans and soft robots, it is necessary to develop a sensing device that can respond to both tactile and non-contact stimuli simultaneously and can distinguish between these two modes in real-time operations.

2. **User-friendly, programming-free teaching methods**: Traditional teaching methods are usually based on contact teaching, suitable for rigid robots, but not for soft robots with infinite degrees of freedom and compliance characteristics. Therefore, a new teaching method is needed to enable non-professional users to easily guide soft robots to complete specific tasks in unstructured environments.

The paper proposes a multimodal flexible sensing interface (FBSS) that combines triboelectric nanogenerators and liquid metal sensors, capable of both tactile and non-contact sensing, and can distinguish between these two modes in real-time. Based on this sensing interface, the researchers developed a "move sensor and teach" method, allowing users to teach soft robots complex 3D movements through coordinated gestures and eye movements. Experimental results show that non-professional users can effectively teach soft robots to complete tasks such as navigating a maze and taking a throat swab within minutes.