Abstract:Abstract Whenever we are addressing a specific object or refer to a certain spatial location, we are using referential or deictic gestures usually accompanied by some verbal description. Particularly, pointing gestures are necessary to dissolve ambiguities in a scene and they are of crucial importance when verbal communication may fail due to environmental conditions or when two persons simply do not speak the same language. With the currently increasing advances of humanoid robots and their future integration in domestic domains, the development of gesture interfaces complementing human–robot interaction scenarios is of substantial interest. The implementation of an intuitive gesture scenario is still challenging because both the pointing intention and the corresponding object have to be correctly recognized in real time. The demand increases when considering pointing gestures in a cluttered environment, as is the case in households. Also, humans perform pointing in many different ways and those variations have to be captured. Research in this field often proposes a set of geometrical computations which do not scale well with the number of gestures and objects and use specific markers or a predefined set of pointing directions. In this paper, we propose an unsupervised learning approach to model the distribution of pointing gestures using a growing-when-required (GWR) network. We introduce an interaction scenario with a humanoid robot and define the so-called ambiguity classes. Our implementation for the hand and object detection is independent of any markers or skeleton models; thus, it can be easily reproduced. Our evaluation comparing a baseline computer vision approach with our GWR model shows that the pointing-object association is well learned even in cases of ambiguities resulting from close object proximity.

Gaze-assisted visual grounding via knowledge distillation for referred object grasping with under-specified object referring

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

InterRep: A Visual Interaction Representation for Robotic Grasping

DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations

Depth-aware gaze-following via auxiliary networks for robotics

KGNet: Knowledge-Guided Networks for Category-Level 6D Object Pose and Size Estimation.

Target-Oriented Object Grasping via Multimodal Human Guidance

Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction

AD-DINO: Attention-Dynamic DINO for Distance-Aware Embodied Reference Understanding

Gaze Target Estimation inspired by Interactive Attention

Knowledge enhanced bottom-up affordance grounding for robotic interaction

End-to-end Visual Grounding Based on Query Text Guidance and Multi-stage Reasoning

Gaze-based, Context-aware Robotic System for Assisted Reaching and Grasping

Deep Multitask Gaze Estimation with a Constrained Landmark-Gaze Model

Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints

Antipodal-Points-aware Dual-decoding Network for Robotic Visual Grasp Detection Oriented to Multi-object Clutter Scenes

Deep Vision Networks for Real-Time Robotic Grasp Detection

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments.

Solving visual object ambiguities when pointing: an unsupervised learning approach

INVIGORATE: Interactive Visual Grounding and Grasping in Clutter

Attentive 3D-Ghost Module for Dynamic Hand Gesture Recognition with Positive Knowledge Transfer