Abstract:With the increasing presence of robots in our daily lives, it is crucial to design interaction interfaces that are natural, easy to use and meaningful for robotic tasks. This is important not only to enhance the user experience but also to increase the task reliability by providing supplementary information. Motivated by this, we propose a multi-modal framework consisting of multiple independent modules. These modules take advantage of multiple sensors (e.g., image, sound, depth) and can be used separately or in combination for effective human–robot collaborative interaction. We identified and implemented four key components of an effective human robot collaborative setting, which included determining object location and pose, extracting intricate information from verbal instructions, resolving user(s) of interest (UOI), and gesture recognition and gaze estimation to facilitate the natural and intuitive interactions. The system uses a feature–detector–descriptor approach for object recognition and a homography-based technique for planar pose estimation and a deep multi-task learning model to extract intricate task parameters from verbal communication. The user of interest (UOI) is detected by estimating the facing state and active speakers. The framework also includes gesture detection and gaze estimation modules, which are combined with a verbal instruction component to form structured commands for robotic entities. Experiments were conducted to assess the performance of these interaction interfaces, and the results demonstrated the effectiveness of the approach.

Real-Time Multi-modal Human-Robot Collaboration Using Gestures and Speech

HRC of intelligent assembly system based on multimodal gesture control

Design of Kinect-Based Human Robot Interaction Systems for A Robocup Middle Size League Soccer Robot

A Human-Robot Collaboration System for Object Handover

A Multi-sensor Gesture Interaction System for Human-robot Cooperation

A Multimodal Emotional Communication Based Humans-Robots Interaction System

Dynamic Hand Gesture-Featured Human Motor Adaptation in Tool Delivery using Voice Recognition

An Attention-Based Deep Learning Approach for Inertial Motion Recognition and Estimation in Human-Robot Collaboration

Gesture recognition using combination of acceleration sensor and images for casual communication between robots and humans

A Novel Gesture Recognition System for Intelligent Interaction with a Nursing-Care Assistant Robot

HRC for dual-robot intelligent assembly system based on multimodal perception

Towards Multi-User Activity Recognition through Facilitated Training Data and Deep Learning for Human-Robot Collaboration Applications

On the Design of Human-Robot Collaboration Gestures

Hands-On Robotics: Enabling Communication Through Direct Gesture Control

Multimodal information fusion for human-robot interaction

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Enabling collaborative assembly between humans and robots using a digital twin system

Online Recognition of Incomplete Gesture Data to Interface Collaborative Robots

A Proposed Set of Communicative Gestures for Human Robot Interaction and an RGB Image-based Gesture Recognizer Implemented in ROS

A Vision-based Human Digital Twin Modelling Approach for Adaptive Human-Robot Collaboration

Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-Robot Interaction