Abstract:With the increasing presence of robots in our daily lives, it is crucial to design interaction interfaces that are natural, easy to use and meaningful for robotic tasks. This is important not only to enhance the user experience but also to increase the task reliability by providing supplementary information. Motivated by this, we propose a multi-modal framework consisting of multiple independent modules. These modules take advantage of multiple sensors (e.g., image, sound, depth) and can be used separately or in combination for effective human–robot collaborative interaction. We identified and implemented four key components of an effective human robot collaborative setting, which included determining object location and pose, extracting intricate information from verbal instructions, resolving user(s) of interest (UOI), and gesture recognition and gaze estimation to facilitate the natural and intuitive interactions. The system uses a feature–detector–descriptor approach for object recognition and a homography-based technique for planar pose estimation and a deep multi-task learning model to extract intricate task parameters from verbal communication. The user of interest (UOI) is detected by estimating the facing state and active speakers. The framework also includes gesture detection and gaze estimation modules, which are combined with a verbal instruction component to form structured commands for robotic entities. Experiments were conducted to assess the performance of these interaction interfaces, and the results demonstrated the effectiveness of the approach.

Utility Functions for Human/Robot Interaction

Human-robot Negotiation of Intentions Based on Virtual Fixtures for Shared Task Execution

Model-based Utility Functions

Understanding the Application of Utility Theory in Robotics and Artificial Intelligence: A Survey

How to be Helpful? Supportive Behaviors and Personalization for Human-Robot Collaboration

Cognitive Architecture for Mutual Modelling

Human and computer interaction in information system design for managing business

Balancing Shared Autonomy with Human-Robot Communication

Utility assessment in automated driving for cooperative human–machine systems

Robots in Human Environments: Basic Autonomous Capabilities

A Human-Robot Co-Manipulation Approach Based on Human Sensorimotor Information

Toward Mutual Trust Modeling in Human-Robot Collaboration

On Human Robot Interaction using Multiple Modes

Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Emotional affordances for human–robot interaction

Human Behavioral Models Using Utility Theory and Prospect Theory

Developing Computational Models of Social Assistance to Guide Socially Assistive Robots

Evaluating physical quantities and learning human utilities from RGBD videos

Shared Autonomy via Variable Impedance Control and Virtual Potential Fields for Encoding Human Demonstration

Inferring Forces And Learning Human Utilities From Videos