Abstract:Enabling robotic manipulation that generalizes to out-of-distribution scenes is a crucial step toward open-world embodied intelligence. For human beings, this ability is rooted in the understanding of semantic correspondence among objects, which naturally transfers the interaction experience of familiar objects to novel ones. Although robots lack such a reservoir of interaction experience, the vast availability of human videos on the Internet may serve as a valuable resource, from which we extract an affordance memory including the contact points. Inspired by the natural way humans think, we propose Robo-ABC: when confronted with unfamiliar objects that require generalization, the robot can acquire affordance by retrieving objects that share visual or semantic similarities from the affordance memory. The next step is to map the contact points of the retrieved objects to the new object. While establishing this correspondence may present formidable challenges at first glance, recent research finds it naturally arises from pre-trained diffusion models, enabling affordance mapping even across disparate object categories. Through the Robo-ABC framework, robots may generalize to manipulate out-of-category objects in a zero-shot manner without any manual annotation, additional training, part segmentation, pre-coded knowledge, or viewpoint restrictions. Quantitatively, Robo-ABC significantly enhances the accuracy of visual affordance retrieval by a large margin of 31.6% compared to state-of-the-art (SOTA) end-to-end affordance models. We also conduct real-world experiments of cross-category object-grasping tasks. Robo-ABC achieved a success rate of 85.7%, proving its capacity for real-world tasks.

Robot′s affordance prediction based on the subtask

Multi-Robot Coordination In Complex Environment With Task And Communication Constraints

Hierarchical Affordance Discovery using Intrinsic Motivation

Utilization of Affordance by Reinforcement Learning Robot

AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions

Humanoid infers Archimedes' principle: understanding physical relations and object affordances through cumulative learning experiences

Promoting Mutual Adaptation in Haptic Negotiation Using Adaptive Virtual Fixture

Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions

Affordance-based Robot Manipulation with Flow Matching

Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation

RAIL: Robot Affordance Imagination with Large Language Models

Human-Aware Robot Task Planning Based on a Hierarchical Task Model

Learning Precise Affordances from Egocentric Videos for Robotic Manipulation

RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation

Information-driven Affordance Discovery for Efficient Robotic Manipulation

Reasoning on Grasp-Action Affordances

Affordance-Centric Policy Learning: Sample Efficient and Generalisable Robot Policy Learning using Affordance-Centric Task Frames

Rediscovering Affordance: A Reinforcement Learning Perspective

Efficient multitask learning with an embodied predictive model for door opening and entry with whole-body control

Toward Proactive Human–Robot Collaborative Assembly: A Multimodal Transfer-Learning-Enabled Action Prediction Approach

Affordance-Based Mobile Robot Navigation Among Movable Obstacles