Abstract:This paper introduces HPOF, a novel deep neural network to reconstruct the 3D human motion from a monocular video. Recently, model-based methods have been proposed to simplify the reconstruction task by estimating several parameters that control a deformable surface model to fit the person in the image. However, learning the parameters from a single image is a highly ill-posed problem, and the process is ultimately data-hungry. Existing 3D datasets are not sufficient, and the usage of 2D in-the-wild datasets is often susceptible to the inadequate precision of manual annotations. To address the above issues, our method yields substantial improvements in two domains. First, we leverage optical flow to supervise the 2D rendered images of predicted SMPL models to learn short-term temporal features. Besides, taking long-term temporal consistency into account, we define a novel temporal encoder based on a dilated convolutional network. The encoder decomposes the learning process of human shape and pose, first guarantees the invariance of the body shape, and then simulates a more reasonable forward kinematics process on this basis to achieve more accurate pose estimation. In addition, an adversarial learning framework is applied to supervise the reconstruction progress in a coarse-grained way. We show that HPOF not only improves the accuracy of 3D poses but ensures the realistic body structure throughout the video. We perform extensive experimentation to demonstrate the superiority of our method and analyze the effectiveness of our model, surpassing other state-of-the-arts.

Optical Flow-based 3D Human Motion Estimation from Monocular Video

Flowpose: Conditional Normalizing Flows for 3D Human Pose and Shape Estimation from Monocular Videos

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

Learning Human Optical Flow

Location Uncertainty Principle: Toward the Definition of Parameter-Free Motion Estimators

Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flows

Learning Multi-Human Optical Flow

ScaleFlow++: Robust and Accurate Estimation of 3D Motion from Video

Physics-based Human Motion Estimation and Synthesis from Videos

High-Fidelity Neural Human Motion Transfer from Monocular Video

MoCo-Flow: Neural Motion Consensus Flow for Dynamic Humans in Stationary Monocular Cameras

Efficient 3D Recovery of Human Motion in Monocular Video

Motion Flow Matching for Human Motion Synthesis and Editing

HPOF:3D Human Pose Recovery from Monocular Video with Optical Flow

Three-dimensional motion estimation by straight-line and endpoint optical flow

DeepFaceFlow: In-the-wild Dense 3D Facial Motion Estimation

Linear Model for 3d Motion Estimation and Shape Reconstruction Based on the Straight-Line Optical Flow

Bootstrapping Human Optical Flow and Pose

Optical Flow in Mostly Rigid Scenes

presented a deep 3 D convolutional architecture that models the flow estimation problem as a “ Voxel 2 Voxel ” prediction problem