Abstract:Multi-frame human pose estimation has long been an appealing and fundamental issue in visual perception. Owing to the frequent rapid motion and pose occlusion in videos, this task is extremely challenging. Current state-of-the-art methods seek to model spatiotemporal features by equally fusing each frame in the local sequence, which weakens the target frame information. In addition, existing approaches usually emphasize more on deep features while ignoring the detailed information implied in the shallow feature maps, resulting in the dropping of crucial features. To address the above problems, we propose an effective framework, namely spatiotemporal learning transformer for video-based human pose estimation (SLT-Pose), which consists of a Personalized Feature Extraction Module (PFEM), Self-feature Refinement Module (SRM), Cross-frame Temporal Learning Module (CTLM) and Disentangled Keypoint Detector (DKD). To be specific, we propose PFEM which extracts and modulates the individual frame features to adapt to the varying human shape, and integrates single-frame features to obtain the spatiotemporal features. We further present SRM to establish global correlation spatial cues on the target frame to attain the refinement feature. Then, a CTLM is designed to search for the information most closely related to the target frame from the spatiotemporal features to intensify the interaction between the target frame and the local sequence, using both the shallow detailed and the deep semantic representations. Finally, we employ DKD to extract the disentangled characteristics of each joint and encode the articulated joint pairs in the human body, promoting the model to reasonably and accurately predict the keypoint heatmaps. Extensive experiments on three huamn motion benchmarks, including PoseTrack2017, PoseTrack2018, and Sub-JHMDB dataset, demonstrate that SLT-Pose plays favorably against state-of-the-art approaches in terms of both objective evaluation and subjective visual performance.

GITPose: going shallow and deeper using vision transformers for human pose estimation

HEViTPose: High-Efficiency Vision Transformer for Human Pose Estimation

MHViTPose: Multiscale Hybrid Vision Transformer for Human Pose Estimation

Bilateral Pose Transformer for Human Pose Estimation.

HRPoseFormer: High-Resolution Transformer for Human Pose Estimation Via Multi-Scale Token Aggregation

PoseGTAC: Graph Transformer Encoder-Decoder with Atrous Convolution for 3D Human Pose Estimation

ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation

VitPose: Multi-view 3D Human Pose Estimation with Vision Transformer

Efficient Vision Transformer for Human Pose Estimation via Patch Selection

Joint graph convolution networks and transformer for human pose estimation in sports technique analysis

ViTPose++: Vision Transformer for Generic Body Pose Estimation

DGT: Dynamic Graph Transformer for Enhanced Processing of Dynamic Joint Sequences in 2D Human Pose Estimation

AggPose: Deep Aggregation Vision Transformer for Infant Pose Estimation

HSTFormer: Hierarchical Spatial-Temporal Transformers for 3D Human Pose Estimation

DGFormer: Dynamic Graph Transformer for 3D Human Pose Estimation

HRPVT: High-Resolution Pyramid Vision Transformer for medium and small-scale human pose estimation

DPIT: Dual-Pipeline Integrated Transformer for Human Pose Estimation

3D Human Pose Estimation with Spatial and Temporal Transformers

IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation

Shift Pose: A Lightweight Transformer-like Neural Network for Human Pose Estimation

Spatiotemporal Learning Transformer for Video-Based Human Pose Estimation