Abstract:Robots must be able to understand their surroundings to perform complex tasks in challenging environments and many of these complex tasks require estimates of physical properties such as friction or weight. Estimating such properties using learning is challenging due to the large amounts of labelled data required for training and the difficulty of updating these learned models online at run time. To overcome these challenges, this paper introduces a novel, multi-modal approach for representing semantic predictions and physical property estimates jointly in a probabilistic manner. By using conjugate pairs, the proposed method enables closed-form Bayesian updates given visual and tactile measurements without requiring additional training data. The efficacy of the proposed algorithm is demonstrated through several hardware experiments. In particular, this paper illustrates that by conditioning semantic classifications on physical properties, the proposed method quantitatively outperforms state-of-the-art semantic classification methods that rely on vision alone. To further illustrate its utility, the proposed method is used in several applications including to represent affordance-based properties probabilistically and a challenging terrain traversal task using a legged robot. In the latter task, the proposed method represents the coefficient of friction of the terrain probabilistically, which enables the use of an on-line risk-aware planner that switches the legged robot from a dynamic gait to a static, stable gait when the expected value of the coefficient of friction falls below a given threshold. Videos of these case studies as well as the open-source C++ and ROS interface can be found at <a class="link-external link-https" href="https://roahmlab.github.io/multimodal_mapping/" rel="external noopener nofollow">this https URL</a>.

You've Got to Feel It To Believe It: Multi-Modal Bayesian Inference for Semantic and Property Prediction

Predictive Visuo-Tactile Interactive Perception Framework for Object Properties Inference

Identifying Terrain Physical Parameters from Vision -- Towards Physical-Parameter-Aware Locomotion and Navigation

Visuo-Tactile based Predictive Cross Modal Perception for Object Exploration in Robotics

Adaptive robot body learning and estimation through predictive coding

Learning to See Physical Properties with Active Sensing Motor Policies

A Vision-Based Two-Stage Framework for Inferring Physical Properties of the Terrain

Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environments

Predict Robot Grasp Outcomes Based On Multi-Modal Information

Multi-modal Probabilistic Prediction of Interactive Behavior Via an Interpretable Model

Learning active tactile perception through belief-space control

Multimodal representation models for prediction and control from partial information

Combining Vision and Tactile Sensation for Video Prediction

Learning Intuitive Physics with Multimodal Generative Models

Predicting Terrain Mechanical Properties in Sight for Planetary Rovers with Semantic Clues

Bridging vision and touch: advancing robotic interaction prediction with self-supervised multimodal learning

Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies

Probabilistic Data Association via Mixture Models for Robust Semantic SLAM

Active inference body perception and action for humanoid robots

Proprioceptive Learning with Soft Polyhedral Networks

Bayesian Nonparametric Object Association for Semantic SLAM