Abstract:State-of-the-art single depth image-based 3D hand pose estimation methods are based on dense predictions, including voxel-to-voxel predictions, point-to-point regression, and pixel-wise estimations. Despite the good performance, those methods have a few issues in nature, such as the poor trade-off between accuracy and efficiency, and plain feature representation learning with local convolutions. In this paper, a novel pixel-wise prediction-based method is proposed to address the above issues. The key ideas are two-fold: (a) explicitly modeling the dependencies among joints and the relations between the pixels and the joints for better local feature representation learning; (b) unifying the dense pixel-wise offset predictions and direct joint regression for end-to-end training. Specifically, we first propose a graph convolutional network (GCN) based joint graph reasoning module to model the complex dependencies among joints and augment the representation capability of each pixel. Then we densely estimate all pixels’ offsets to joints in both image plane and depth space and calculate the joints’ positions by a weighted average over all pixels’ predictions, totally discarding the complex post-processing operations. The proposed model is implemented with an efficient 2D fully convolutional network (FCN) backbone and has only about 1.4M parameters. Extensive experiments on multiple 3D hand pose estimation benchmarks demonstrate that the proposed method achieves new state-of-the-art accuracy while running very efficiently with around a speed of 110 fps on a single NVIDIA 1080Ti GPU (This work was supported in part by the National Natural Science Foundation of China under Grants 61976095, in part by the Science and Technology Planning Project of Guangdong Province under Grant 2018B030323026. This work was also partially supported by the Academy of Finland.). The code is available at https://github.com/fanglinpu/JGR-P2O.

Joint-wise 2D to 3D lifting for hand pose estimation from a single RGB image

A2J-Transformer: Anchor-to-Joint Transformer Network for 3D Interacting Hand Pose Estimation from a Single RGB Image

Joint Hand-Object 3D Reconstruction From a Single Image With Cross-Branch Feature Fusion

Joint Hand and Object Pose Estimation from a Single RGB Image Using High-level 2D Constraints

Joint Hand-Object Pose Estimation with Differentiably-Learned Physical Contact Point Analysis

A 3D Hand Attitude Estimation Method for Fixed Hand Posture Based on Dual-View RGB Images

Lifting by Image -- Leveraging Image Cues for Accurate 3D Human Pose Estimation

A hybrid network for estimating 3D interacting hand pose from a single RGB image

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

Robust Estimation of 3D Human Poses from a Single Image

Simultaneous 3D Hand Detection and Pose Estimation Using Single Depth Images

Using a single RGB frame for real time 3D hand pose estimation in the wild

End-to-End Weakly-Supervised Single-Stage Multiple 3d Hand Mesh Reconstruction from a Single Rgb Image

Graph-Based CNNs With Self-Supervised Module for 3D Hand Pose Estimation From Monocular RGB

3D Hand Pose Estimation via Regularized Graph Representation Learning

JGR-P2O: Joint Graph Reasoning Based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Image

Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

Model-based 3D Hand Reconstruction via Self-Supervised Learning

Hand Pose Estimation via Latent 2.5D Heatmap Regression