Abstract:With the increasing presence of robots in our daily lives, it is crucial to design interaction interfaces that are natural, easy to use and meaningful for robotic tasks. This is important not only to enhance the user experience but also to increase the task reliability by providing supplementary information. Motivated by this, we propose a multi-modal framework consisting of multiple independent modules. These modules take advantage of multiple sensors (e.g., image, sound, depth) and can be used separately or in combination for effective human–robot collaborative interaction. We identified and implemented four key components of an effective human robot collaborative setting, which included determining object location and pose, extracting intricate information from verbal instructions, resolving user(s) of interest (UOI), and gesture recognition and gaze estimation to facilitate the natural and intuitive interactions. The system uses a feature–detector–descriptor approach for object recognition and a homography-based technique for planar pose estimation and a deep multi-task learning model to extract intricate task parameters from verbal communication. The user of interest (UOI) is detected by estimating the facing state and active speakers. The framework also includes gesture detection and gaze estimation modules, which are combined with a verbal instruction component to form structured commands for robotic entities. Experiments were conducted to assess the performance of these interaction interfaces, and the results demonstrated the effectiveness of the approach.

A General Framework of Task Understanding for Tour-Guide Robots in Exhibition Environments

The Integrated Design of an Interactive Service Robot for Shanghai World Expo 2010

A Comprehensive Review of Task Understanding of Command-Triggered Execution of Tasks for Service Robots

A Multi-modal Behavior Planning Framework for Guide Robot.

Heterogeneous Context-Aware Robots Providing a Personalized Building Tour

A Universal Self-Adaption Workspace Mapping Method for Human–Robot Interaction Using Kinect Sensor Data

Providers-Clients-Robots: Framework for spatial-semantic planning for shared understanding in human-robot interaction

Cross-modal Task Understanding and Execution of Voice-fingertip Reading Instruction by Using Small Family Service Robotic

TeleAware Robot: Designing Awareness-augmented Telepresence Robot for Remote Collaborative Locomotion

Developing high-level cognitive functions for service robots

A User Interface for Sense-making of the Reasoning Process while Interacting with Robots

Network Guide Robot System Proactively Initiating Interaction with Humans Based on Their Local and Global Behaviors

A Framework to Co-Optimize Robot Exploration and Task Planning in Unknown Environments

GUI3DXBot: An Interactive Software Tool for a Tour-Guide Mobile Robot

A novel task-oriented framework for dual-arm robotic assembly task

A Framework for Intelligent Service Environments Based on Middleware and General Purpose Task Planner

A social planning and navigation for tour-guide robot in human environment

A service robot architecture supporting individualized coordination

Social Robot Navigation and Comfortable Following: A Novel Knowledge-Based Robot-Pedestrian Interaction Model with Self Learning Strategy.

Natural Language Understanding for Chinese-based Service Robots

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition