Abstract:Multi-frame human pose estimation has long been an appealing and fundamental issue in visual perception. Owing to the frequent rapid motion and pose occlusion in videos, this task is extremely challenging. Current state-of-the-art methods seek to model spatiotemporal features by equally fusing each frame in the local sequence, which weakens the target frame information. In addition, existing approaches usually emphasize more on deep features while ignoring the detailed information implied in the shallow feature maps, resulting in the dropping of crucial features. To address the above problems, we propose an effective framework, namely spatiotemporal learning transformer for video-based human pose estimation (SLT-Pose), which consists of a Personalized Feature Extraction Module (PFEM), Self-feature Refinement Module (SRM), Cross-frame Temporal Learning Module (CTLM) and Disentangled Keypoint Detector (DKD). To be specific, we propose PFEM which extracts and modulates the individual frame features to adapt to the varying human shape, and integrates single-frame features to obtain the spatiotemporal features. We further present SRM to establish global correlation spatial cues on the target frame to attain the refinement feature. Then, a CTLM is designed to search for the information most closely related to the target frame from the spatiotemporal features to intensify the interaction between the target frame and the local sequence, using both the shallow detailed and the deep semantic representations. Finally, we employ DKD to extract the disentangled characteristics of each joint and encode the articulated joint pairs in the human body, promoting the model to reasonably and accurately predict the keypoint heatmaps. Extensive experiments on three huamn motion benchmarks, including PoseTrack2017, PoseTrack2018, and Sub-JHMDB dataset, demonstrate that SLT-Pose plays favorably against state-of-the-art approaches in terms of both objective evaluation and subjective visual performance.

MSRT: multi-scale representation transformer for regression-based human pose estimation

Adaptively Fusing Complete Multi-resolution Features for Human Pose Estimation.

Poseur: Direct Human Pose Regression with Transformers.

TFPose: Direct Human Pose Estimation with Transformers

HRPoseFormer: High-Resolution Transformer for Human Pose Estimation Via Multi-Scale Token Aggregation

MHViTPose: Multiscale Hybrid Vision Transformer for Human Pose Estimation

Gated Region-Refine Pose Transformer for Human Pose Estimation.

End-to-End Multi-Person Pose Estimation with Transformers.

U-shaped Spatial–temporal Transformer Network for 3D Human Pose Estimation

Spatiotemporal Learning Transformer for Video-Based Human Pose Estimation

Rethinking on Multi-Stage Networks for Human Pose Estimation

A Multi-scale Recalibrated Approach for 3D Human Pose Estimation.

A Transformer-Based Network for Human Pose Estimation using Millimeter Wave Radar Data

Multi-Scale Supervised Network for Human Pose Estimation

MRSAPose: Multi-level Routing Sparse Attention for Multi-Person Pose Estimation

A Lightweight Context-Aware Feature Transformer Network for Human Pose Estimation

Towards Precise 3D Human Pose Estimation with Multi-Perspective Spatial-Temporal Relational Transformers

A Multi-Level Network for Human Pose Estimation

Bilateral Pose Transformer for Human Pose Estimation.

3D Human Pose Estimation with Spatial and Temporal Transformers

A Multi-stage Feature Fusion Network for Human Pose Estimation