Abstract:Multi-frame human pose estimation has long been an appealing and fundamental issue in visual perception. Owing to the frequent rapid motion and pose occlusion in videos, this task is extremely challenging. Current state-of-the-art methods seek to model spatiotemporal features by equally fusing each frame in the local sequence, which weakens the target frame information. In addition, existing approaches usually emphasize more on deep features while ignoring the detailed information implied in the shallow feature maps, resulting in the dropping of crucial features. To address the above problems, we propose an effective framework, namely spatiotemporal learning transformer for video-based human pose estimation (SLT-Pose), which consists of a Personalized Feature Extraction Module (PFEM), Self-feature Refinement Module (SRM), Cross-frame Temporal Learning Module (CTLM) and Disentangled Keypoint Detector (DKD). To be specific, we propose PFEM which extracts and modulates the individual frame features to adapt to the varying human shape, and integrates single-frame features to obtain the spatiotemporal features. We further present SRM to establish global correlation spatial cues on the target frame to attain the refinement feature. Then, a CTLM is designed to search for the information most closely related to the target frame from the spatiotemporal features to intensify the interaction between the target frame and the local sequence, using both the shallow detailed and the deep semantic representations. Finally, we employ DKD to extract the disentangled characteristics of each joint and encode the articulated joint pairs in the human body, promoting the model to reasonably and accurately predict the keypoint heatmaps. Extensive experiments on three huamn motion benchmarks, including PoseTrack2017, PoseTrack2018, and Sub-JHMDB dataset, demonstrate that SLT-Pose plays favorably against state-of-the-art approaches in terms of both objective evaluation and subjective visual performance.

A Multi-Person Pose Estimation with LSTM for Video Stream

LSTM Pose Machines.

Multi-Person Pose Estimation Using Bounding Box Constraint and LSTM.

Deep Dual Consecutive Network for Human Pose Estimation

Live Stream Temporally Embedded 3D Human Body Pose and Shape Estimation

3D Human pose estimation from video via multi-scale multi-level spatial temporal features

VH3D-LSFM: Video-Based Human 3D Pose Estimation with Long-Term and Short-Term Pose Fusion Mechanism.

Exploring Temporal Consistency for Human Pose Estimation in Videos

Multi-Person Pose Estimation with Enhanced Channel-wise and Spatial Information

A Deconvolutional Bottom-up Deep Network for Multi-Person Pose Estimation.

3D Human Pose and Shape Estimation from Video

Spatiotemporal Learning Transformer for Video-Based Human Pose Estimation

Rethinking on Multi-Stage Networks for Human Pose Estimation

Joint Human Detection and Head Pose Estimation Via Multistream Networks for RGB-D Videos

Human Pose Estimation Based on Lightweight Multi-Scale Coordinate Attention

A Multi-Level Network for Human Pose Estimation

Multi-Person Articulated Tracking With Spatial and Temporal Embeddings

Shape and Pose Estimation for Closely Interacting Persons Using Multi-view Images.

Multi-Scale Supervised Network for Human Pose Estimation

AdaptivePose++: A Powerful Single-Stage Network for Multi-Person Pose Regression

Multi-person 3D pose estimation from unlabelled data