Abstract:Video-based 3D human pose and shape estimations are evaluated by intra-frame accuracy and inter-frame smoothness. Although these two metrics are responsible for different ranges of temporal consistency, existing state-of-the-art methods treat them as a unified problem and use monotonous modeling structures (e.g., RNN or attention-based block) to design their networks. However, using a single kind of modeling structure is difficult to balance the learning of short-term and long-term temporal correlations, and may bias the network to one of them, leading to undesirable predictions like global location shift, temporal inconsistency, and insufficient local details. To solve these problems, we propose to structurally decouple the modeling of long-term and short-term correlations in an end-to-end framework, Global-to-Local Transformer (GLoT). First, a global transformer is introduced with a Masked Pose and Shape Estimation strategy for long-term modeling. The strategy stimulates the global transformer to learn more inter-frame correlations by randomly masking the features of several frames. Second, a local transformer is responsible for exploiting local details on the human mesh and interacting with the global transformer by leveraging cross-attention. Moreover, a Hierarchical Spatial Correlation Regressor is further introduced to refine intra-frame estimations by decoupled global-local representation and implicit kinematic constraints. Our GLoT surpasses previous state-of-the-art methods with the lowest model parameters on popular benchmarks, i.e., 3DPW, MPI-INF-3DHP, and Human3.6M. Codes are available at https://github.com/sxl142/GLoT.

Deterministic Model Fitting by Local-Neighbor Preservation and Global-Residual Optimization

GORFLM: Globally Optimal Robust Fitting for Linear Model

Latent Semantic Consensus for Deterministic Geometric Model Fitting

Superpixel-Guided Two-View Deterministic Geometric Model Fitting

Superpixel-Based Two-View Deterministic Fitting For Multiple-Structure Data

Motion Consistency Guided Robust Geometric Model Fitting with Severe Outliers

A Hierarchical Voting Scheme For Robust Geometric Model Fitting

Local Optimized and Scalable Frame-to-model SLAM

Robust Outlier Removal Using Penalized Linear Regression in Multiview Geometry

Induced Subgraph Searching for Geometric Model Fitting

Efficient Robust Model Fitting for Multistructure Data Using Global Greedy Search

Hypergraph Optimization for Multi-structural Geometric Model Fitting

Message Passing On The Two-Layer Network For Geometric Model Fitting

A Novel Robust Point Cloud Fitting Algorithm Based on Nonlinear Gauss–Helmert Model

Learning to find reliable correspondences with local neighborhood consensus

A Novel Robust Model Fitting Approach Towards Multiple-Structure Data Segmentation

Robust Geometric Model Fitting Based on Iterative Hypergraph Construction and Partition

Globally Optimal Linear Model Fitting with Unit-Norm Constraint

Conceptual Space Based Model Fitting for Multi-Structure Data

Second-order Proximity Guided Sampling Consensus for Robust Model Fitting

Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation