Abstract:Multi-view 3D human pose estimation is naturally superior to single view one, benefiting from more comprehensive information provided by images of multiple views. The information includes camera poses, 2D/3D human poses, and 3D geometry. However, the accurate annotation of these information is hard to obtain, making it challenging to predict accurate 3D human pose from multi-view images. To deal with this issue, we propose a fully self-supervised framework, named cascaded multi-view aggregating network (CMANet), to construct a canonical parameter space to holistically integrate and exploit multi-view information. In our framework, the multi-view information is grouped into two categories: 1) intra-view information , 2) inter-view information. Accordingly, CMANet consists of two components: intra-view module (IRV) and inter-view module (IEV). IRV is used for extracting initial camera pose and 3D human pose of each view; IEV is to fuse complementary pose information and cross-view 3D geometry for a final 3D human pose. To facilitate the aggregation of the intra- and inter-view, we define a canonical parameter space, depicted by per-view camera pose and human pose and shape parameters ($\theta$ and $\beta$) of SMPL model, and propose a two-stage learning procedure. At first stage, IRV learns to estimate camera pose and view-dependent 3D human pose supervised by confident output of an off-the-shelf 2D keypoint detector. At second stage, IRV is frozen and IEV further refines the camera pose and optimizes the 3D human pose by implicitly encoding the cross-view complement and 3D geometry constraint, achieved by jointly fitting predicted multi-view 2D keypoints. The proposed framework, modules, and learning strategy are demonstrated to be effective by comprehensive experiments and CMANet is superior to state-of-the-art methods in extensive quantitative and qualitative analysis.

Self-Supervised Human Pose Based Multi-Camera Video Synchronization

Ivs-Net: Learning Human View Synthesis from Internet Videos

Efficient Multi-person Hierarchical 3D Pose Estimation for Autonomous Driving

Single-Frame based Deep View Synchronization for Unsynchronized Multi-Camera Surveillance

Unsupervised Universal Hierarchical Multi-Person 3D Pose Estimation for Natural Scenes

Subframe-Level Synchronization in Multi-Camera System Using Time-Calibrated Video

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

Unveiling the Power of Self-supervision for Multi-view Multi-human Association and Tracking

3D Human pose estimation from video via multi-scale multi-level spatial temporal features

Multi-person Multi-Camera Tracking for Live Stream Videos Based on Improved Motion Model and Matching Cascade

Self-supervised Keypoint Correspondences for Multi-Person Pose Estimation and Tracking in Videos

3d Pose Detection Of Closely Interactive Humans Using Multi-View Cameras

Event Sequence Matching-Based Multi-Camera Video Synchronization

Geometry-Driven Self-Supervised Method for 3D Human Pose Estimation

Weakly Supervised 3D Multi-Person Pose Estimation for Large-Scale Scenes Based on Monocular Camera and Single LiDAR

Self-supervised Multi-view Multi-Human Association and Tracking

SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation

Deep Dual Consecutive Network for Human Pose Estimation

Novel View Synthesis of Human Interactions from Sparse Multi-view Videos

Advancing Video Synchronization with Fractional Frame Analysis: Introducing a Novel Dataset and Model

Self-learning Canonical Space for Multi-view 3D Human Pose Estimation