Abstract:Affordance grounding aims to localize the interaction regions for the manipulated objects in the scene image according to given instructions. A critical challenge in affordance grounding is that the embodied agent should understand human instructions and analyze which tools in the environment can be used, as well as how to use these tools to accomplish the instructions. Most recent works primarily supports simple action labels as input instructions for localizing affordance regions, failing to capture complex human objectives. Moreover, these approaches typically identify affordance regions of only a single object in object-centric images, ignoring the object context and struggling to localize affordance regions of multiple objects in complex scenes for practical applications. To address this concern, for the first time, we introduce a new task of affordance grounding based on natural language instructions, extending it from previously using simple labels for complex human instructions. For this new task, we propose a new framework, WorldAfford. We design a novel Affordance Reasoning Chain-of-Thought Prompting to reason about affordance knowledge from LLMs more precisely and logically. Subsequently, we use SAM and CLIP to localize the objects related to the affordance knowledge in the image. We identify the affordance regions of the objects through an affordance region localization module. To benchmark this new task and validate our framework, an affordance grounding dataset, LLMaFF, is constructed. We conduct extensive experiments to verify that WorldAfford performs state-of-the-art on both the previous AGD20K and the new LLMaFF dataset. In particular, WorldAfford can localize the affordance regions of multiple objects and provide an alternative when objects in the environment cannot fully match the given instruction.

Grounded Affordance from Exocentric View

Learning Affordance Grounding from Exocentric Images

Grounding 3D Object Affordance from 2D Interactions in Images

Grounding 3D Scene Affordance From Egocentric Interactions

Learning Object Affordance with Contact and Grasp Generation

Weakly Supervised Multimodal Affordance Grounding for Egocentric Images

INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding

AffordanceLLM: Grounding Affordance from Vision Language Models

Learning Visual Affordance Grounding from Demonstration Videos

WorldAfford: Affordance Grounding based on Natural Language Instructions

Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding

Visual-Geometric Collaborative Guidance for Affordance Learning

GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding

Affordance Grounding from Demonstration Video to Target Image

Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)

PartAfford: Part-level Affordance Discovery from 3D Objects

O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning

Learning Precise Affordances from Egocentric Videos for Robotic Manipulation

Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions

Text-driven Affordance Learning from Egocentric Vision

Grounded Human-Object Interaction Hotspots from Video