Abstract:In this letter, a paradigm for the classification and manipulation of previously unseen objects is established and demonstrated through a real example of chairs. We present a novel robot manipulation method, guided by the understanding of object stability, perceptibility, and affordance, which allows the robot to prepare previously unseen and randomly oriented chairs for a teddy bear to sit on. Specifically, the robot encounters an unknown object and first reconstructs a complete 3D model from perceptual data via active and autonomous manipulation. By inserting this model into a physical simulator (i.e., the robot's "imagination"), the robot assesses whether the object is a chair and determines how to reorient it properly to be used, i.e., how to reorient it to an upright and accessible pose. If the object is classified as a chair, the robot reorients the object to this pose and seats the teddy bear onto the chair. The teddy bear is a proxy for an elderly person, hospital patient, or child. Experiment results show that our method achieves a high success rate on the real robot task of chair preparation. Also, it outperforms several baseline methods on the task of upright pose prediction for chairs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to enable a robot to understand and manipulate previously unseen objects, especially chairs, so as to make them suitable for an agent (such as a teddy bear) to sit on. Specifically, the paper proposes a new robot manipulation method. By understanding the stability, perceptibility and functional operability of objects, this method enables the robot to prepare a suitable sitting posture on a chair in an unknown and random direction. This process includes: 1. **Object Reconstruction**: When a robot encounters an unknown object, it first reconstructs a complete 3D model from the perception data through active and autonomous operations. Since parts of the object will be occluded when it is placed on a plane, the robot needs to find a stable pose to expose the occluded parts and fuse the images captured from different perspectives to accurately reconstruct the object. 2. **Physical Simulation Evaluation**: Insert the reconstructed model into a physical simulator (i.e., the robot's "imagination") to evaluate whether the object is a chair and how to re - orient it to a suitable position for use, that is, how to adjust it to an upright and accessible posture. 3. **Chair Recognition and Adjustment**: If the object is classified as a chair, the robot will adjust it to this posture and let the teddy bear sit on the chair. The teddy bear, as an agent, represents the elderly, hospital patients or children. The paper verifies through experiments that its method has a high success rate in real - robot tasks, especially in chair - preparation tasks, and is superior to several baseline methods in predicting the upright posture of chairs. In addition, the paper also explores how to explore the functional operability of objects through simulating physical interactions, which is a method called "robot imagination" proposed in the paper.

Prepare the Chair for the Bear! Robot Imagination of Sitting Affordance to Reorient Previously Unseen Chairs

Put the Bear on the Chair! Intelligent Robot Interaction with Previously Unseen Chairs via Robot Imagination

Is That a Chair? Imagining Affordances Using Simulations of an Articulated Human Body

Vision-Based Categorical Object Pose Estimation and Manipulation.

Automatic Sitting Pose Generation for Ergonomic Ratings of Chairs

RAIL: Robot Affordance Imagination with Large Language Models

Reactive Human-to-Robot Handovers of Arbitrary Objects

FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation

Task-driven Perception and Manipulation for Constrained Placement of Unknown Objects

Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control

Learning Articulated Constraints from a One-Shot Demonstration for Robot Manipulation Planning

Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects

Learning Robotic Manipulation through Visual Planning and Acting

ReorientBot: Learning Object Reorientation for Specific-Posed Placement

Object Wake-up: 3-D Object Reconstruction, Animation, and in-situ Rendering from a Single Image

Manipulation-Oriented Object Perception in Clutter through Affordance Coordinate Frames

Safe and Efficient Robot Manipulation: Task-Oriented Environment Modeling and Object Pose Estimation

Automating Robotic Furniture with A Collaborative Vision-based Sensing Scheme

Ergonomic-driven Geometric Exploration and Reshaping

Object-Centric Dexterous Manipulation from Human Motion Data

CHAIRS: Towards Full-Body Articulated Human-Object Interaction