Abstract:Abstract Whenever we are addressing a specific object or refer to a certain spatial location, we are using referential or deictic gestures usually accompanied by some verbal description. Particularly, pointing gestures are necessary to dissolve ambiguities in a scene and they are of crucial importance when verbal communication may fail due to environmental conditions or when two persons simply do not speak the same language. With the currently increasing advances of humanoid robots and their future integration in domestic domains, the development of gesture interfaces complementing human–robot interaction scenarios is of substantial interest. The implementation of an intuitive gesture scenario is still challenging because both the pointing intention and the corresponding object have to be correctly recognized in real time. The demand increases when considering pointing gestures in a cluttered environment, as is the case in households. Also, humans perform pointing in many different ways and those variations have to be captured. Research in this field often proposes a set of geometrical computations which do not scale well with the number of gestures and objects and use specific markers or a predefined set of pointing directions. In this paper, we propose an unsupervised learning approach to model the distribution of pointing gestures using a growing-when-required (GWR) network. We introduce an interaction scenario with a humanoid robot and define the so-called ambiguity classes. Our implementation for the hand and object detection is independent of any markers or skeleton models; thus, it can be easily reproduced. Our evaluation comparing a baseline computer vision approach with our GWR model shows that the pointing-object association is well learned even in cases of ambiguities resulting from close object proximity.

Ambiguities in Spatial Language Understanding in Situated Human Robot Dialogue.

Resolving Positional Ambiguity in Dialogues by Vision-Language Models for Robot Navigation

Treat Robots As Humans? Perspective Choice in Human-Human and Human-Robot Spatial Language Interaction.

Towards Mediating Shared Perceptual Basis in Situated Dialogue.

Perspective-Corrected Spatial Referring Expression Generation for Human-Robot Interaction

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities

Do People Regard Robots As Human-Like Social Partners? Evidence from Perspective-Taking in Spatial Descriptions.

Collaborative Effort Towards Common Ground in Situated Human-Robot Dialogue.

Speaking Your Language: Spatial Relationships in Interpretable Emergent Communication

Semantically-Driven Disambiguation for Human-Robot Interaction

Towards Understanding Language through Perception in Situated Human-Robot Interaction: From Word Grounding to Grammar Induction

Towards Situated Dialogue: Revisiting Referring Expression Generation.

SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction

Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction

Understanding Spatial Relations through Multiple Modalities

Strategies for human-driven robot comprehension of spatial descriptions by older adults in a robot fetch task

Situational Awareness Matters in 3D Vision Language Reasoning

Human-Robot Dialogue Annotation for Multi-Modal Common Ground

Solving visual object ambiguities when pointing: an unsupervised learning approach