Abstract:From an early age, humans learn to develop an intuition for the physical nature of the objects around them by using exploratory behaviors. Such exploration provides observations of how objects feel, sound, look, and move as a result of actions applied on them. Previous works in robotics have shown that robots can also use such behaviors (e.g., lifting, pressing, shaking) to infer object properties that camera input alone cannot detect. Such learned representations are specific to each individual robot and cannot currently be transferred directly to another robot with different sensors and actions. Moreover, sensor failure can cause a robot to lose a specific sensory modality which may prevent it from using perceptual models that require it as input. To address these limitations, we propose a framework for knowledge transfer across behaviors and sensory modalities such that: (1) knowledge can be transferred from one or more robots to another, and, (2) knowledge can be transferred from one or more sensory modalities to another. We propose two different models for transfer based on variational auto-encoders and encoder-decoder networks. The main hypothesis behind our approach is that if two or more robots share multi-sensory object observations of a shared set of objects, then those observations can be used to establish mappings between multiple features spaces, each corresponding to a combination of an exploratory behavior and a sensory modality. We evaluate our approach on a category recognition task using a dataset in which a robot used 9 behaviors, coupled with 4 sensory modalities, performed multiple times on 100 objects. The results indicate that sensorimotor knowledge about objects can be transferred both across behaviors and across sensory modalities, such that a new robot (or the same robot, but with a different set of sensors) can bootstrap its category recognition models without having to exhaustively explore the full set of objects.

Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies

Cross-Tool and Cross-Behavior Perceptual Knowledge Transfer for Grounded Object Recognition

Inferring Object Properties from Human Interaction and Transferring Them to New Motions

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

Predictive Visuo-Tactile Interactive Perception Framework for Object Properties Inference

MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception

Visualizing Robot Intent for Object Handovers with Augmented Reality

Reactive Human-to-Robot Handovers of Arbitrary Objects

Know Thyself: Transferable Visual Control Policies Through Robot-Awareness

Learning Human-to-Robot Dexterous Handovers for Anthropomorphic Hand

Transfer: Cross Modality Knowledge Transfer using Adversarial Networks -- A Study on Gesture Recognition

Multiperspective Teaching of Unknown Objects via Shared-gaze-based Multimodal Human-Robot Interaction

Human-Humanoid Robots Cross-Embodiment Behavior-Skill Transfer Using Decomposed Adversarial Learning from Demonstration

Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers

Learning Object Properties Using Robot Proprioception via Differentiable Robot-Object Interaction

Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning

Visuo-Tactile based Predictive Cross Modal Perception for Object Exploration in Robotics

Object-Independent Human-to-Robot Handovers Using Real Time Robotic Vision

NeuralGrasps: Learning Implicit Representations for Grasps of Multiple Robotic Hands

Learning Human-to-Robot Handovers from Point Clouds.

Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignment