Abstract:Open containers, i.e., containers without covers, are an important and ubiquitous class of objects in human life. In this letter, we propose a novel method for robots to "imagine" the open containability affordance of a previously unseen object via physical simulations. We implement our imagination method on a UR5 manipulator. The robot autonomously scans the object with an RGB-D camera. The scanned 3D model is used for open containability imagination which quantifies the open containability affordance by physically simulating dropping particles onto the object and counting how many particles are retained in it. This quantification is used for open-container vs. non-open-container binary classification (hereafter referred to as open container classification). If the object is classified as an open container, the robot further imagines pouring into the object, again using physical simulations, to obtain the pouring position and orientation for real robot autonomous pouring. We evaluate our method on open container classification and autonomous pouring of granular material on a dataset containing 130 previously unseen objects with 57 object categories. Although our proposed method uses only 11 objects for simulation calibration (training), its open container classification aligns well with human judgements. In addition, our method endows the robot with the capability to autonomously pour into the 55 containers in the dataset with a very high success rate. We also compare to a deep learning method. Results show that our method achieves the same performance as the deep learning method on open container classification and outperforms it on autonomous pouring. Moreover, our method is fully explainable.

The Sound of Water: Inferring Physical Properties from Pouring Liquids

ASLiquid: Non-Intrusive Liquid Counterfeit Identification with Your Earphones

Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring

PourIt!: Weakly-supervised Liquid Perception from a Single Image for Visual Closed-Loop Robotic Pouring

See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content

Understanding Dynamic Tactile Sensing for Liquid Property Estimation

Robust Robotic Pouring using Audition and Haptics

To Stir or Not to Stir: Online Estimation of Liquid Properties for Pouring Actions

Understanding Dynamic Auditory and Tactile Perception for Water Filling Level Estimation

LiquidListener: Supporting Ubiquitous Liquid Volume Sensing via Singing Sounds

Learning to Pour

Can I Pour into It? Robot Imagining Open Containability Affordance of Previously Unseen Objects via Physical Simulations

Audiovisual Crossmodal Correspondence between Bubbles’ Size and Pouring Sounds’ Pitch in Carbonated Beverages

Self-supervised Transparent Liquid Segmentation for Robotic Pouring

Robot Gaining Accurate Pouring Skills through Self-Supervised Learning and Generalization

Visual-Tactile Sensing for Real-time Liquid Volume Estimation in Grasping

Hearing temperatures: employing machine learning for elucidating the cross-modal perception of thermal properties through audition

Vision-Language Model-based Physical Reasoning for Robot Liquid Perception

Seeing liquids from visual motion

Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

Visual to Sound: Generating Natural Sound for Videos in the Wild