Abstract:Inferring human-scene contact (HSC) is the first step toward understanding how humans interact with their surroundings. While detecting 2D human-object interaction (HOI) and reconstructing 3D human pose and shape (HPS) have enjoyed significant progress, reasoning about 3D human-scene contact from a single image is still challenging. Existing HSC detection methods consider only a few types of predefined contact, often reduce body and scene to a small number of primitives, and even overlook image evidence. To predict human-scene contact from a single image, we address the limitations above from both data and algorithmic perspectives. We capture a new dataset called RICH for "Real scenes, Interaction, Contact and Humans." RICH contains multiview outdoor/indoor video sequences at 4K resolution, ground-truth 3D human bodies captured using markerless motion capture, 3D body scans, and high resolution 3D scene scans. A key feature of RICH is that it also contains accurate vertex-level contact labels on the body. Using RICH, we train a network that predicts dense body-scene contacts from a single RGB image. Our key insight is that regions in contact are always occluded so the network needs the ability to explore the whole image for evidence. We use a transformer to learn such non-local relationships and propose a new Body-Scene contact TRansfOrmer (BSTRO). Very few methods explore 3D contact; those that do focus on the feet only, detect foot contact as a post-processing step, or infer contact from body pose without looking at the scene. To our knowledge, BSTRO is the first method to directly estimate 3D body-scene contact from a single image. We demonstrate that BSTRO significantly outperforms the prior art. The code and dataset are available at <a class="link-external link-https" href="https://rich.is.tue.mpg.de" rel="external noopener nofollow">this https URL</a>.

Automatic Human Scene Interaction through Contact Estimation and Motion Adaptation

Revisit Human-Scene Interaction via Space Occupancy

The Design and Implementation of Natural Human-Robot Interaction System Based on Kinect Sensor

Contact-aware Human Motion Forecasting

A Universal Self-Adaption Workspace Mapping Method for Human–Robot Interaction Using Kinect Sensor Data

HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR

Reconstructing Three-Dimensional Models of Interacting Humans

Human-centric Application in Cyber-Physical System: an Inertial-based Motion Capture and Recognition System

Unified Human-Scene Interaction via Prompted Chain-of-Contacts

Unsupervised Universal Hierarchical Multi-Person 3D Pose Estimation for Natural Scenes

Kinematics-based 3D Human-Object Interaction Reconstruction from Single View

HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes

CG-HOI: Contact-Guided 3D Human-Object Interaction Generation

Capturing and Inferring Dense Full-Body Human-Scene Contact

Ins-HOI: Instance Aware Human-Object Interactions Recovery

Contact-Aware Retargeting of Skinned Motion

Full-body Motion Capture for Multiple Closely Interacting Persons.

Two-Person Interaction Augmentation with Skeleton Priors

Scaling Up Dynamic Human-Scene Interaction Modeling

Learning Complex 3D Human Self-Contact

HSPACE: Synthetic Parametric Humans Animated in Complex Environments