Language-Grounded Control for Coordinated Robot Motion and Speech

Ravi Tejwani,Chengyuan Ma,Paco Gomez-Paz,Paolo Bonato,H. Harry Asada

DOI: https://doi.org/10.48550/arXiv.2305.05456

2023-10-10

Abstract:Recent advancements have enabled human-robot collaboration through physical assistance and verbal guidance. However, limitations persist in coordinating robots' physical motions and speech in response to real-time changes in human behavior during collaborative contact tasks. We first derive principles from analyzing physical therapists' movements and speech during patient exercises. These principles are translated into control objectives to: 1) guide users through trajectories, 2) control motion and speech pace to align completion times with varying user cooperation, and 3) dynamically paraphrase speech along the trajectory. We then propose a Language Controller that synchronizes motion and speech, modulating both based on user cooperation. Experiments with 12 users show the Language Controller successfully aligns motion and speech compared to baselines. This provides a framework for fluent human-robot collaboration.

Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in human - robot collaborative tasks, how can the robot naturally coordinate its physical actions with voice guidance to adapt to the real - time changes of human behavior. Specifically, existing robots have made certain progress in providing physical assistance and language guidance, but they still have limitations in responding to changes in human behavior in real - time, especially in tasks requiring close physical contact. These limitations are mainly manifested in the robot's inability to dynamically adjust the speed of its movement and voice to match the cooperation level of different users, resulting in less smooth and natural human - robot collaboration.

To overcome these problems, the paper proposes a new control framework - "Language Controller", which can dynamically adjust the movement speed and voice speed of the robot according to the real - time reactions of the user, thereby achieving more natural human - robot collaboration. Specific goals include:
1. Guide the user to complete the predetermined trajectory while adjusting the movement speed according to the user's cooperation level.
2. Control the voice speed of the robot to match the movement speed and be able to adapt to different cooperation levels of the user.
3. Dynamically rephrase the voice instructions so that the voice length matches the movement time, for example, using longer sentences when the movement is slower and shorter sentences when the movement is faster.

Through experimental verification, this controller can successfully achieve the above - mentioned goals in tests with different users, improving the fluency and naturalness of human - robot collaboration.