Abstract:Motion model based video coding approach, which employs sparse sets of keypoints instead of dense optical flows, can efficiently compress videos at ultra-low bitrates. Such schemes obtain notable performance gains over traditional video codecs in face-centric scenarios, such as video conferencing. However, due to the high complexity of human poses, there is still a lack of research on motion model based human body video coding, especially in the case of large pose variations. In order to overcome this limitation, we present a thin-plate spline motion model based portrait video compression framework oriented to adaptive pose processing. Firstly a more flexible thin-plate spline transformation rather than simple affine transformation is adopted for motion estimation, since the nonlinear property allows representing more complex motions. Meanwhile, spatial constraints are incorporated into the keypoint detector to generate keypoints that are more consistent with the human poses, thus obtaining more accurate optical flow. In addition, a motion intensity evaluation module is designed at the encoder side to dynamically evaluate the inter-frame motion intensity. Adaptive Reference Frame Selection algorithm is then further devised at the decoder side to adaptively select the reconstruction scheme for different intensities of portrait motion. Finally, a multi-frame reconstruction module is introduced for large pose variations to improve the consistency of human pose and subjective quality. The experimental results demonstrate that compared to the state-of-the-art video coding standard Versatile Video Coding and existing motion model based compression techniques, our proposed scheme can better cope with large pose variation scenarios and outperforms in both objective and subjective quality at the similar bitrate with higher temporal consistency.

Fast Human Pose Estimation in Compressed Videos

Model-based portrait video compression with spatial constraint and adaptive pose processing

Fast Online Upper Body Pose Estimation from Video.

Live Stream Temporally Embedded 3D Human Body Pose and Shape Estimation

3D Human pose estimation from video via multi-scale multi-level spatial temporal features

An Optimization Based Framework for Human Pose Estimation

Robust 3D Human Pose Estimation from Single Images or Video Sequences

Pose-Driven Compression for Dynamic 3D Human Via Human Prior Models.

Towards Accurate Human Pose Estimation in Videos of Crowded Scenes

Deep Dual Consecutive Network for Human Pose Estimation

A Fast, Robust Optical Flow Estimation Method for Compressed Video

An Effective 3D Human Pose Estimation Method Based on Dilated Convolutions for Videos.

3D Human Pose Estimation with Single Image and Inertial Measurement Unit (IMU) Sequence

Leveraging Compressed Frame Sizes For Ultra-Fast Video Classification

Multiview human pose estimation with unconstrained motions

Kinematic-aware Hierarchical Attention Network for Human Pose Estimation in Videos

Decomposed Human Motion Prior for Video Pose Estimation via Adversarial Training

Fast Human Motion reconstruction from sparse inertial measurement units considering the human shape

MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video

Imocap: Motion Capture from Internet Videos

FastHand: Fast monocular hand pose estimation on embedded systems