Abstract:We present Multi-view Pose transformer (MvP) for estimating multi-person 3D poses from multi-view images. Instead of estimating 3D joint locations from costly volumetric representation or reconstructing the per-person 3D pose from multiple detected 2D poses as in previous methods, MvP directly regresses the multi-person 3D poses in a clean and efficient way, without relying on intermediate tasks. Specifically, MvP represents skeleton joints as learnable query embeddings and let them progressively attend to and reason over the multi-view information from the input images to directly regress the actual 3D joint locations. To improve the accuracy of such a simple pipeline, MvP presents a hierarchical scheme to concisely represent query embeddings of multi-person skeleton joints and introduces an input-dependent query adaptation approach. Further, MvP designs a novel geometrically guided attention mechanism, called projective attention, to more precisely fuse the cross-view information for each joint. MvP also introduces a RayConv operation to integrate the view-dependent camera geometry into the feature representations for augmenting the projective attention. We show experimentally that our MvP model outperforms the state-of-the-art methods on several benchmarks while being much more efficient. Notably, it achieves 92.3% AP25 on the challenging Panoptic dataset, improving upon the previous best approach [36] by 9.8%. MvP is general and also extendable to recovering human mesh represented by the SMPL model, thus useful for modeling multi-person body shapes. Code and models are available at <a class="link-external link-https" href="https://github.com/sail-sg/mvp" rel="external noopener nofollow">this https URL</a>.

Respective Volumetric Heatmap Autoencoder for Multi-Person 3D Pose Estimation

Efficient Multi-person Hierarchical 3D Pose Estimation for Autonomous Driving

Exploring Severe Occlusion: Multi-Person 3D Pose Estimation with Gated Convolution.

X-HRNet: Towards Lightweight Human Pose Estimation with Spatially Unidimensional Self-Attention

3D Human Pose Estimation in Motion Based on Multi-Stage Regression

3D Human Pose Estimation Based on Volumetric Joint Coordinates

Unsupervised Universal Hierarchical Multi-Person 3D Pose Estimation for Natural Scenes

3D Associative Embedding: Multi-View 3D Human Pose Estimation in Crowded Scenes.

A Scale‐sensitive Heatmap Representation for Multi‐person Pose Estimation

3D Human Pose Estimation by Depth Map

VTP: Volumetric Transformer for Multi-view Multi-person 3D Pose Estimation

Multi-Person Pose Estimation with Accurate Heatmap Regression and Greedy Association

Faster VoxelPose: Real-time 3D Human Pose Estimation by Orthographic Projection.

Attention Residual Network with 3D convolutional neural network for 3D Human Pose Estimation.

HEMlets PoSh: Learning Part-Centric Heatmap Triplets for 3D Human Pose and Shape Estimation

A Joint Relationship Aware Neural Network for Single-Image 3D Human Pose Estimation

Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting

Learning a Robust Part-Aware Monocular 3D Human Pose Estimator via Neural Architecture Search

Towards Locality Similarity Preserving to 3D Human Pose Estimation.

Multi-Person 3D Pose Estimation from Multi-View Uncalibrated Depth Cameras

Direct Multi-view Multi-person 3D Pose Estimation