Abstract:From an early age, humans learn to develop an intuition for the physical nature of the objects around them by using exploratory behaviors. Such exploration provides observations of how objects feel, sound, look, and move as a result of actions applied on them. Previous works in robotics have shown that robots can also use such behaviors (e.g., lifting, pressing, shaking) to infer object properties that camera input alone cannot detect. Such learned representations are specific to each individual robot and cannot currently be transferred directly to another robot with different sensors and actions. Moreover, sensor failure can cause a robot to lose a specific sensory modality which may prevent it from using perceptual models that require it as input. To address these limitations, we propose a framework for knowledge transfer across behaviors and sensory modalities such that: (1) knowledge can be transferred from one or more robots to another, and, (2) knowledge can be transferred from one or more sensory modalities to another. We propose two different models for transfer based on variational auto-encoders and encoder-decoder networks. The main hypothesis behind our approach is that if two or more robots share multi-sensory object observations of a shared set of objects, then those observations can be used to establish mappings between multiple features spaces, each corresponding to a combination of an exploratory behavior and a sensory modality. We evaluate our approach on a category recognition task using a dataset in which a robot used 9 behaviors, coupled with 4 sensory modalities, performed multiple times on 100 objects. The results indicate that sensorimotor knowledge about objects can be transferred both across behaviors and across sensory modalities, such that a new robot (or the same robot, but with a different set of sensors) can bootstrap its category recognition models without having to exhaustively explore the full set of objects.

Open-Ended Fine-Grained 3D Object Categorization by Combining Shape and Texture Features in Multiple Colorspaces

Look Further to Recognize Better: Learning Shared Topics and Category-Specific Dictionaries for Open-Ended 3D Object Recognition

3D_DEN: Open-ended 3D Object Recognition using Dynamically Expandable Networks

Investigating the Importance of Shape Features, Color Constancy, Color Spaces and Similarity Measures in Open-Ended 3D Object Recognition

Fine-grained 3D object recognition: an approach and experiments

A Multisensory Learning Architecture for Rotation-invariant Object Recognition

Lifelong ensemble learning based on multiple representations for few-shot object recognition

Automatic Object Searching and Behavior Learning for Mobile Robots in Unstructured Environment by Deep Belief Networks.

Learning visual object models on a robot using context and appearance cues

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructions

Move and the Robot Will Learn: Vision-based Autonomous Learning of Object Models.

Using Scale Coordination and Semantic Information for Robust 3-D Object Recognition by a Service Robot

Learning Few-Shot Object Placement with Intra-Category Transfer

Learning Spatial Object Localization from Vision on a Humanoid Robot

Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifiers

Learning Canonical 3D Object Representation for Fine-Grained Recognition

Vision-based Autonomous Learning of Object Models on a Mobile Robot

3D Object Recognition By Corresponding and Quantizing Neural 3D Scene Representations

Open-Set Object Recognition Using Mechanical Properties During Interaction