Abstract:Predicting human motion based on a sequence of past observations is crucial for various applications in robotics and computer vision. Currently, this problem is typically addressed by training deep learning models using some of the most well-known 3D human motion datasets widely used in the community. However, these datasets generally do not consider how humans behave and move when a robot is nearby, leading to a data distribution different from the real distribution of motion that robots will encounter when collaborating with humans. Additionally, incorporating contextual information related to the interactive task between the human and the robot, as well as information on the human willingness to collaborate with the robot, can improve not only the accuracy of the predicted sequence but also serve as a useful tool for robots to navigate through collaborative tasks successfully. In this research, we propose a deep learning architecture that predicts both 3D human body motion and human intention for collaborative tasks. The model employs a multi-head attention mechanism, taking into account human motion and task context as inputs. The resulting outputs include the predicted motion of the human body and the inferred human intention. We have validated this architecture in two different tasks: collaborative object handover and collaborative grape harvesting. While the architecture remains the same for both tasks, the inputs differ. In the handover task, the architecture considers human motion, robot end effector, and obstacle positions as inputs. Additionally, the model can be conditioned on the desired intention to tailor the output motion accordingly. To assess the performance of the collaborative handover task, we conducted a user study to evaluate human perception of the robot's sociability, naturalness, security, and comfort. This evaluation was conducted by comparing the robot's behavior when it utilized the prediction in its planner versus when it did not. Furthermore, we also applied the model to a collaborative grape harvesting task. By integrating human motion prediction and human intention inference, our architecture shows promising results in enhancing the capabilities of robots in collaborative scenarios. The model's flexibility allows it to handle various tasks with different inputs, making it adaptable to real-world applications.

Early Estimation of User's Intention of Tele-Operation Using Object Affordance and Hand Motion in a Dual First-Person Vision

Human-robot Negotiation of Intentions Based on Virtual Fixtures for Shared Task Execution

A Haptic Shared Control Architecture for Tracking of a Moving Object.

A Sensor Glove Based on Inertial Measurement Unit for Robot Teleoperetion.

Visualizing Robot Intent for Object Handovers with Augmented Reality

An Agile Large-Workspace Teleoperation Interface Based on Human Arm Motion and Force Estimation

Human–Robot Shared Control Based on Locally Weighted Intent Prediction for a Teleoperated Hydraulic Manipulator System

Real-time estimation of human motion intention for a hand rehabilitation robot with forearm EMG signals

FMG- and RNN-Based Estimation of Motor Intention of Upper-Limb Motion in Human-Robot Collaboration

An Intent-based Task-aware Shared Control Framework for Intuitive Hands Free Telemanipulation

Space Robot Teleoperation Based on Active Vision

Fast human motion prediction for human-robot collaboration with wearable interfaces

Constraint-Aware Intent Estimation for Dynamic Human-Robot Object Co-Manipulation

Human-Machine Interface for Remote Training of Robot Tasks

Virtual Reality Based Tactile Sensing Enhancements for Bilateral Teleoperation System with In-Hand Manipulation

Mixed-Reality-Enhanced Human–Robot Interaction with an Imitation-Based Mapping Approach for Intuitive Teleoperation of a Robotic Arm-Hand System

Enhancing Robotic Collaborative Tasks Through Contextual Human Motion Prediction and Intention Inference

Combining haptics and inertial motion capture to enhance remote control of a dual-arm robot

Learning Human-arm Reaching Motion Using IMU in Human-Robot Collaboration

Towards Haptic-Based Dual-Arm Manipulation

Arm Robot: AR-Enhanced Embodied Control and Visualization for Intuitive Robot Arm Manipulation