Abstract:Affordance grounding, a task to ground (i.e., localize) action possibility region in objects, which faces the challenge of establishing an explicit link with object parts due to the diversity of interactive affordance. Human has the ability that transform the various exocentric interactions to invariant egocentric affordance so as to counter the impact of interactive diversity. To empower an agent with such ability, this paper proposes a task of affordance grounding from exocentric view, i.e., given exocentric human-object interaction and egocentric object images, learning the affordance knowledge of the object and transferring it to the egocentric image using only the affordance label as supervision. To this end, we devise a cross-view knowledge transfer framework that extracts affordance-specific features from exocentric interactions and enhances the perception of affordance regions by preserving affordance correlation. Specifically, an Affordance Invariance Mining module is devised to extract specific clues by minimizing the intra-class differences originated from interaction habits in exocentric images. Besides, an Affordance Co-relation Preserving strategy is presented to perceive and localize affordance by aligning the co-relation matrix of predicted results between the two views. Particularly, an affordance grounding dataset named AGD20K is constructed by collecting and labeling over 20K images from 36 affordance categories. Experimental results demonstrate that our method outperforms the representative models in terms of objective metrics and visual quality. Code: <a class="link-external link-http" href="http://github.com/lhc1224/Cross-View-AG" rel="external noopener nofollow">this http URL</a>.

Learning Visual Affordance Grounding from Demonstration Videos

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

Learning Object Affordance with Contact and Grasp Generation

Affordance Grounding from Demonstration Video to Target Image

Visual-Geometric Collaborative Guidance for Affordance Learning

Learning Affordance Grounding from Exocentric Images

Grounding 3D Object Affordance from 2D Interactions in Images

Intention-Related Natural Language Grounding Via Object Affordance Detection and Intention Semantic Extraction

Learning Precise Affordances from Egocentric Videos for Robotic Manipulation

Grounded Affordance from Exocentric View

AffordanceLLM: Grounding Affordance from Vision Language Models

Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)

LAC-Net: Linear-Fusion Attention-Guided Convolutional Network for Accurate Robotic Grasping Under the Occlusion

Leverage Interactive Affinity for Affordance Learning

Weakly Supervised Multimodal Affordance Grounding for Egocentric Images

Grounding 3D Scene Affordance From Egocentric Interactions

Grounded Human-Object Interaction Hotspots from Video

Fed-HANet: Federated Visual Grasping Learning for Human Robot Handovers

Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding

3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding