Neural monocular 3D human motion capture with physical awareness

Soshi Shimada,Vladislav Golyanik,Weipeng Xu,Patrick Pérez,Christian Theobalt

DOI: https://doi.org/10.1145/3476576.3476640

IF: 6.2

2021-08-01

ACM Transactions on Graphics

Abstract:ACM Transactions on Graphics, <a href="https://dl.acm.org/toc/tog/2021/40/4">Volume 40, Issue 4</a>, Page 1-15, August 2021. <br/>

computer science, software engineering

What problem does this paper attempt to address?

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

Buzhen Huang,Liang Pan,Yuan Yang,Jingyi Ju,Yangang Wang

DOI: https://doi.org/10.48550/arXiv.2203.14065

2022-03-26

Abstract:Due to the visual ambiguity, purely kinematic formulations on monocular human motion capture are often physically incorrect, biomechanically implausible, and can not reconstruct accurate interactions. In this work, we focus on exploiting the high-precision and non-differentiable physics simulator to incorporate dynamical constraints in motion capture. Our key-idea is to use real physical supervisions to train a target pose distribution prior for sampling-based motion control to capture physically plausible human motion. To obtain accurate reference motion with terrain interactions for the sampling, we first introduce an interaction constraint based on SDF (Signed Distance Field) to enforce appropriate ground contact modeling. We then design a novel two-branch decoder to avoid stochastic error from pseudo ground-truth and train a distribution prior with the non-differentiable physics simulator. Finally, we regress the sampling distribution from the current state of the physical character with the trained prior and sample satisfied target poses to track the estimated reference motion. Qualitative and quantitative results show that we can obtain physically plausible human motion with complex terrain interactions, human shape variations, and diverse behaviors. More information can be found at~\url{<a class="link-external link-https" href="https://www.yangangwang.com/papers/HBZ-NM-2022-03.html" rel="external noopener nofollow">this https URL</a>}

Computer Vision and Pattern Recognition
Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

Yu Du,Yongkang Wong,Yonghao Liu,Feilin Han,Yilin Gui,Zhen Wang,Mohan S. Kankanhalli,Weidong Geng

DOI: https://doi.org/10.1007/978-3-319-46493-0_2

2016-01-01

Abstract:The recovery of 3D human pose with monocular camera is an inherently ill-posed problem due to the large number of possible projections from the same 2D image to 3D space. Aimed at improving the accuracy of 3D motion reconstruction, we introduce the additional built-in knowledge, namely height-map, into the algorithmic scheme of reconstructing the 3D pose/motion under a single-view calibrated camera. Our novel proposed framework consists of two major contributions. Firstly, the RGB image and its calculated height-map are combined to detect the landmarks of 2D joints with a dual-stream deep convolution network. Secondly, we formulate a new objective function to estimate 3D motion from the detected 2D joints in the monocular image sequence, which reinforces the temporal coherence constraints on both the camera and 3D poses. Experiments with HumanEva, Human3.6M, and MCAD dataset validate that our method outperforms the state-of-the-art algorithms on both 2D joints localization and 3D motion recovery. Moreover, the evaluation results on HumanEva indicates that the performance of our proposed single-view approach is comparable to that of the multiview deep learning counterpart.
MonoCap: Monocular Human Motion Capture Using a CNN Coupled with a Geometric Prior

Xiaowei Zhou,Menglong Zhu,Georgios Pavlakos,Spyridon Leonardos,Konstantinos G. Derpanis,Kostas Daniilidis

DOI: https://doi.org/10.1109/tpami.2018.2816031

IF: 23.6

2017-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Recovering 3D full-body human pose is a challenging problem with many applications. It has been successfully addressed by motion capture systems with body worn markers and multiple cameras. In this paper, we address the more challenging case of not only using a single camera but also not leveraging markers: going directly from 2D appearance to 3D geometry. Deep learning approaches have shown remarkable abilities to discriminatively learn 2D appearance features. The missing piece is how to integrate 2D, 3D, and temporal information to recover 3D geometry and account for the uncertainties arising from the discriminative model. We introduce a novel approach that treats 2D joint locations as latent variables whose uncertainty distributions are given by a deep fully convolutional neural network. The unknown 3D poses are modeled by a sparse representation and the 3D parameter estimates are realized via an Expectation-Maximization algorithm, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Extensive evaluation on benchmark datasets shows that the proposed approach achieves greater accuracy over state-of-the-art baselines. Notably, the proposed approach does not require synchronized 2D-3D data for training and is applicable to "in-the-wild" images, which is demonstrated with the MPII dataset.
NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action

Kuan-Chieh Wang,Zhenzhen Weng,Maria Xenochristou,Joao Pedro Araujo,Jeffrey Gu,C. Karen Liu,Serena Yeung

DOI: https://doi.org/10.48550/arXiv.2212.13660

2022-12-28

Abstract:The task of reconstructing 3D human motion has wideranging applications. The gold standard Motion capture (MoCap) systems are accurate but inaccessible to the general public due to their cost, hardware and space constraints. In contrast, monocular human mesh recovery (HMR) methods are much more accessible than MoCap as they take single-view videos as inputs. Replacing the multi-view Mo- Cap systems with a monocular HMR method would break the current barriers to collecting accurate 3D motion thus making exciting applications like motion analysis and motiondriven animation accessible to the general public. However, performance of existing HMR methods degrade when the video contains challenging and dynamic motion that is not in existing MoCap datasets used for training. This reduces its appeal as dynamic motion is frequently the target in 3D motion recovery in the aforementioned applications. Our study aims to bridge the gap between monocular HMR and multi-view MoCap systems by leveraging information shared across multiple video instances of the same action. We introduce the Neural Motion (NeMo) field. It is optimized to represent the underlying 3D motions across a set of videos of the same action. Empirically, we show that NeMo can recover 3D motion in sports using videos from the Penn Action dataset, where NeMo outperforms existing HMR methods in terms of 2D keypoint detection. To further validate NeMo using 3D metrics, we collected a small MoCap dataset mimicking actions in Penn Action,and show that NeMo achieves better 3D reconstruction compared to various baselines.

Computer Vision and Pattern Recognition
MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency

Mingyi Shi,Kfir Aberman,Andreas Aristidou,Taku Komura,Dani Lischinski,Daniel Cohen-Or,Baoquan Chen

DOI: https://doi.org/10.1145/3407659

2020-06-22

Abstract:We introduce MotioNet, a deep neural network that directly reconstructs the motion of a 3D human skeleton from monocular <a class="link-external link-http" href="http://video.While" rel="external noopener nofollow">this http URL</a> previous methods rely on either rigging or inverse kinematics (IK) to associate a consistent skeleton with temporally coherent joint rotations, our method is the first data-driven approach that directly outputs a kinematic skeleton, which is a complete, commonly used, motion representation. At the crux of our approach lies a deep neural network with embedded kinematic priors, which decomposes sequences of 2D joint positions into two separate attributes: a single, symmetric, skeleton, encoded by bone lengths, and a sequence of 3D joint rotations associated with global root positions and foot contact labels. These attributes are fed into an integrated forward kinematics (FK) layer that outputs 3D positions, which are compared to a ground truth. In addition, an adversarial loss is applied to the velocities of the recovered rotations, to ensure that they lie on the manifold of natural joint rotations. The key advantage of our approach is that it learns to infer natural joint rotations directly from the training data, rather than assuming an underlying model, or inferring them from joint positions using a data-agnostic IK solver. We show that enforcing a single consistent skeleton along with temporally coherent joint rotations constrains the solution space, leading to a more robust handling of self-occlusions and depth ambiguities.

Computer Vision and Pattern Recognition,Graphics,Machine Learning
Human Motion Tracking with Less Constraint of Initial Posture from a Single RGB-D Sensor

Chen Liu,Anna Wang,Chunguang Bu,Wenhui Wang,Haijing Sun

DOI: https://doi.org/10.3390/s21093029

IF: 3.9

2021-04-26

Sensors

Abstract:High-quality and complete human motion 4D reconstruction is of great significance for immersive VR and even human operation. However, it has inevitable self-scanning constraints, and tracking under monocular settings also has strict restrictions. In this paper, we propose a human motion capture system combined with human priors and performance capture that only uses a single RGB-D sensor. To break the self-scanning constraint, we generated a complete mesh only using the front view input to initialize the geometric capture. In order to construct a correct warping field, most previous methods initialize their systems in a strict way. To maintain high fidelity while increasing the easiness of the system, we updated the model while capturing motion. Additionally, we blended in human priors in order to improve the reliability of model warping. Extensive experiments demonstrated that our method can be used more comfortably while maintaining credible geometric warping and remaining free of self-scanning constraints.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Real-time 3D neural facial animation from binocular video

Chen Cao,Vasu Agrawal,Fernando De La Torre,Lele Chen,Jason Saragih,Tomas Simon,Yaser Sheikh

DOI: https://doi.org/10.1145/3476576.3476645

IF: 6.2

2021-08-01

ACM Transactions on Graphics

Abstract:ACM Transactions on Graphics, <a href="https://dl.acm.org/toc/tog/2021/40/4">Volume 40, Issue 4</a>, Page 1-17, August 2021. <br/>

computer science, software engineering
Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video

Wen-Li Wei,Jen-Chun Lin,Tyng-Luh Liu,Hong-Yuan Mark Liao

DOI: https://doi.org/10.48550/arXiv.2203.08534

2022-03-16

Abstract:Learning to capture human motion is essential to 3D human pose and shape estimation from monocular video. However, the existing methods mainly rely on recurrent or convolutional operation to model such temporal information, which limits the ability to capture non-local context relations of human motion. To address this problem, we propose a motion pose and shape network (MPS-Net) to effectively capture humans in motion to estimate accurate and temporally coherent 3D human pose and shape from a video. Specifically, we first propose a motion continuity attention (MoCA) module that leverages visual cues observed from human motion to adaptively recalibrate the range that needs attention in the sequence to better capture the motion continuity dependencies. Then, we develop a hierarchical attentive feature integration (HAFI) module to effectively combine adjacent past and future feature representations to strengthen temporal correlation and refine the feature representation of the current frame. By coupling the MoCA and HAFI modules, the proposed MPS-Net excels in estimating 3D human pose and shape in the video. Though conceptually simple, our MPS-Net not only outperforms the state-of-the-art methods on the 3DPW, MPI-INF-3DHP, and Human3.6M benchmark datasets, but also uses fewer network parameters. The video demos can be found at <a class="link-external link-https" href="https://mps-net.github.io/MPS-Net/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Neural Capture of Animatable 3D Human from Monocular Video.

Gusi Te,Xiu Li,Xiao Li,Jinglu Wang,Wei Hu,Yan Lu

DOI: https://doi.org/10.1007/978-3-031-20068-7_16

2022-01-01

Abstract:We present a novel paradigm of building an animatable 3D human representation from a monocular video input, such that it can be rendered in any unseen poses and views. Our method is based on a dynamic Neural Radiance Field (NeRF) rigged by a mesh-based parametric 3D human model serving as a geometry proxy. Previous methods usually rely on multi-view videos or accurate 3D geometry information as additional inputs; besides, most methods suffer from degraded quality when generalized to unseen poses. We identify that the key to generalization is a good input embedding for querying dynamic NeRF: A good input embedding should define an injective mapping in the full volumetric space, guided by surface mesh deformation under pose variation. Based on this observation, we propose to embed the input query with its relationship to local surface regions spanned by a set of geodesic nearest neighbors on mesh vertices. By including both position and relative distance information, our embedding defines a distance-preserved deformation mapping and generalizes well to unseen poses. To reduce the dependency on additional inputs, we first initialize per-frame 3D meshes using off-the-shelf tools and then propose a pipeline to jointly optimize NeRF and refine the initial mesh. Extensive experiments show our method can synthesize plausible human rendering results under unseen poses and views.
Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

Yuxiao Zhou,Marc Habermann,Weipeng Xu,Ikhsanul Habibie,Christian Theobalt,Feng Xu

DOI: https://doi.org/10.48550/arXiv.2003.09572

2022-03-11

Abstract:We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy. This is enabled by a new learning based architecture designed such that it can make use of all the sources of available hand training data: image data with either 2D or 3D annotations, as well as stand-alone 3D animations without corresponding image data. It features a 3D hand joint detection module and an inverse kinematics module which regresses not only 3D joint positions but also maps them to joint rotations in a single feed-forward pass. This output makes the method more directly usable for applications in computer vision and graphics compared to only regressing 3D joint positions. We demonstrate that our architectural design leads to a significant quantitative and qualitative improvement over the state of the art on several challenging benchmarks. Our model is publicly available for future research.

Computer Vision and Pattern Recognition
Monocular 3D Human Pose Markerless Systems for Gait Assessment

Xuqi Zhu,Issam Boukhennoufa,Bernard Liew,Cong Gao,Wangyang Yu,Klaus D McDonald-Maier,Xiaojun Zhai

DOI: https://doi.org/10.3390/bioengineering10060653

2023-05-26

Abstract:Gait analysis plays an important role in the fields of healthcare and sports sciences. Conventional gait analysis relies on costly equipment such as optical motion capture cameras and wearable sensors, some of which require trained assessors for data collection and processing. With the recent developments in computer vision and deep neural networks, using monocular RGB cameras for 3D human pose estimation has shown tremendous promise as a cost-effective and efficient solution for clinical gait analysis. In this paper, a markerless human pose technique is developed using motion captured by a consumer monocular camera (800 × 600 pixels and 30 FPS) for clinical gait analysis. The experimental results have shown that the proposed post-processing algorithm significantly improved the original human pose detection model (BlazePose)'s prediction performance compared to the gold-standard gait signals by 10.7% using the MoVi dataset. In addition, the predicted T2 score has an excellent correlation with ground truth (r = 0.99 and y = 0.94x + 0.01 regression line), which supports that our approach can be a potential alternative to the conventional marker-based solution to assist the clinical gait assessment.
Delving into Motion-Aware Matching for Monocular 3D Object Tracking

Kuan-Chih Huang,Ming-Hsuan Yang,Yi-Hsuan Tsai

2023-08-23

Abstract:Recent advances of monocular 3D object detection facilitate the 3D multi-object tracking task based on low-cost camera sensors. In this paper, we find that the motion cue of objects along different time frames is critical in 3D multi-object tracking, which is less explored in existing monocular-based approaches. In this paper, we propose a motion-aware framework for monocular 3D MOT. To this end, we propose MoMA-M3T, a framework that mainly consists of three motion-aware components. First, we represent the possible movement of an object related to all object tracklets in the feature space as its motion features. Then, we further model the historical object tracklet along the time frame in a spatial-temporal perspective via a motion transformer. Finally, we propose a motion-aware matching module to associate historical object tracklets and current observations as final tracking results. We conduct extensive experiments on the nuScenes and KITTI datasets to demonstrate that our MoMA-M3T achieves competitive performance against state-of-the-art methods. Moreover, the proposed tracker is flexible and can be easily plugged into existing image-based 3D object detectors without re-training. Code and models are available at <a class="link-external link-https" href="https://github.com/kuanchihhuang/MoMA-M3T" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Decaf: Monocular Deformation Capture for Face and Hand Interactions

Soshi Shimada,Vladislav Golyanik,Patrick Pérez,Christian Theobalt

2023-10-13

Abstract:Existing methods for 3D tracking from monocular RGB videos predominantly consider articulated and rigid objects. Modelling dense non-rigid object deformations in this setting remained largely unaddressed so far, although such effects can improve the realism of the downstream applications such as AR/VR and avatar communications. This is due to the severe ill-posedness of the monocular view setting and the associated challenges. While it is possible to naively track multiple non-rigid objects independently using 3D templates or parametric 3D models, such an approach would suffer from multiple artefacts in the resulting 3D estimates such as depth ambiguity, unnatural intra-object collisions and missing or implausible deformations. Hence, this paper introduces the first method that addresses the fundamental challenges depicted above and that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system. As a pivotal step in its creation, we process the reconstructed raw 3D shapes with position-based dynamics and an approach for non-uniform stiffness estimation of the head tissues, which results in plausible annotations of the surface deformations, hand-face contact regions and head-hand positions. At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations. Our final 3D hand and face reconstructions are realistic and more plausible compared to several baselines applicable in our setting, both quantitatively and qualitatively. <a class="link-external link-https" href="https://vcai.mpi-inf.mpg.de/projects/Decaf" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics,Human-Computer Interaction
Accurate realtime full-body motion capture using a single depth camera

Xiaolin Wei,Peizhao Zhang,Jinxiang Chai

DOI: https://doi.org/10.1145/2366145.2366207

IF: 6.2

2012-11-01

ACM Transactions on Graphics

Abstract:We present a fast, automatic method for accurately capturing full-body motion data using a single depth camera. At the core of our system lies a realtime registration process that accurately reconstructs 3D human poses from single monocular depth images, even in the case of significant occlusions. The idea is to formulate the registration problem in a Maximum A Posteriori (MAP) framework and iteratively register a 3D articulated human body model with monocular depth cues via linear system solvers. We integrate depth data, silhouette information, full-body geometry, temporal pose priors, and occlusion reasoning into a unified MAP estimation framework. Our 3D tracking process, however, requires manual initialization and recovery from failures. We address this challenge by combining 3D tracking with 3D pose detection. This combination not only automates the whole process but also significantly improves the robustness and accuracy of the system. Our whole algorithm is highly parallel and is therefore easily implemented on a GPU. We demonstrate the power of our approach by capturing a wide range of human movements in real time and achieve state-of-the-art accuracy in our comparison against alternative systems such as Kinect [2012].

computer science, software engineering
UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality

Christopher D. Twigg,Mathias Parger,Chengcheng Tang,Yijing Li,M. Steinberger,Lingling Tao,Yuanlu Xu,Robert Wang

DOI: https://doi.org/10.1109/TVCG.2021.3085407

IF: 5.2

2020-11-12

IEEE Transactions on Visualization and Computer Graphics

Abstract:Tracking body and hand motions in 3D space is essential for social and self-presence in augmented and virtual environments. Unlike the popular 3D pose estimation setting, the problem is often formulated as egocentric tracking based on embodied perception (e.g., egocentric cameras, handheld sensors). In this article, we propose a new data-driven framework for egocentric body tracking, targeting challenges of omnipresent occlusions in optimization-based methods (e.g., inverse kinematics solvers). We first collect a large-scale motion capture dataset with both body and finger motions using optical markers and inertial sensors. This dataset focuses on social scenarios and captures ground truth poses under self-occlusions and body-hand interactions. We then simulate the occlusion patterns in head-mounted camera views on the captured ground truth using a ray casting algorithm and learn a deep neural network to infer the occluded body parts. Our experiments show that our method is able to generate high-fidelity embodied poses by applying the proposed method to the task of real-time egocentric body tracking, finger motion synthesis, and 3-point inverse kinematics.

Medicine,Computer Science,Engineering
Towards robust 3d reconstruction of human motion from monocular video

Cheng Chen,Yueting Zhuang,Jun Xiao

DOI: https://doi.org/10.1007/11941354_61

2006-01-01

Abstract:A robust, adaptive system for reconstructing 3D human motion from monocular video is presented. Our system takes a model based approach. To save computation time, we fit the skeleton instead of full body model to the silhouette. End sites positions identified from the silhouettes serve as an extra constraint. The alleviation of computational burden then makes the use of simulated annealing practical to get rid of local minima. The identifiable end sites also serve as the criterion to judge the reconstruction reliability of single frames. According to different reliabilities, the video is segmented into sections, which are reconstructed using different strategies. Our system is robust, getting rid of error accumulation in tracking, and adaptive, being able to tell the user when and where more information is needed.
Enhanced Spatio-Temporal Context for Temporally Consistent Robust 3D Human Motion Recovery from Monocular Videos

Sushovan Chanda,Amogh Tiwari,Lokender Tiwari,Brojeshwar Bhowmick,Avinash Sharma,Hrishav Barua

2023-11-20

Abstract:Recovering temporally consistent 3D human body pose, shape and motion from a monocular video is a challenging task due to (self-)occlusions, poor lighting conditions, complex articulated body poses, depth ambiguity, and limited availability of annotated data. Further, doing a simple perframe estimation is insufficient as it leads to jittery and implausible results. In this paper, we propose a novel method for temporally consistent motion estimation from a monocular video. Instead of using generic ResNet-like features, our method uses a body-aware feature representation and an independent per-frame pose and camera initialization over a temporal window followed by a novel spatio-temporal feature aggregation by using a combination of self-similarity and self-attention over the body-aware features and the perframe initialization. Together, they yield enhanced spatiotemporal context for every frame by considering remaining past and future frames. These features are used to predict the pose and shape parameters of the human body model, which are further refined using an LSTM. Experimental results on the publicly available benchmark data show that our method attains significantly lower acceleration error and outperforms the existing state-of-the-art methods over all key quantitative evaluation metrics, including complex scenarios like partial occlusion, complex poses and even relatively low illumination.

Computer Vision and Pattern Recognition
Real-time Physics-based Motion Capture with Sparse Sensors

Sheldon Andrews,Ivan Huerta,Taku Komura,Leonid Sigal,Kenny Mitchell

DOI: https://doi.org/10.1145/2998559.2998564

2016-01-01

Abstract:We propose a framework for real-time tracking of humans using sparse multi-modal sensor sets, including data obtained from optical markers and inertial measurement units. A small number of sensors leaves the performer unencumbered by not requiring dense coverage of the body. An inverse dynamics solver and physics-based body model are used, ensuring physical plausibility by computing joint torques and contact forces. A prior model is also used to give an improved estimate of motion of internal joints. The behaviour of our tracker is evaluated using several black box motion priors. We show that our system can track and simulate a wide range of dynamic movements including bipedal gait, ballistic movements such as jumping, and interaction with the environment. The reconstructed motion has low error and appears natural. As both the internal forces and contacts are obtained with high credibility, it is also useful for human movement analysis.
4D Human Body Capture from Egocentric Video via 3D Scene Grounding

Miao Liu,Dexin Yang,Yan Zhang,Zhaopeng Cui,James M. Rehg,Siyu Tang

DOI: https://doi.org/10.48550/arXiv.2011.13341

2021-10-16

Abstract:We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture. To address those challenges, we propose a simple yet effective optimization-based approach that leverages 2D observations of the entire video sequence and human-scene interaction constraint to estimate second-person human poses, shapes, and global motion that are grounded on the 3D environment captured from the egocentric view. We conduct detailed ablation studies to validate our design choice. Moreover, we compare our method with the previous state-of-the-art method on human motion capture from monocular video, and show that our method estimates more accurate human-body poses and shapes under the challenging egocentric setting. In addition, we demonstrate that our approach produces more realistic human-scene interaction.

Computer Vision and Pattern Recognition
SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos

Xin Chen,Anqi Pang,Wei Yang,Yuexin Ma,Lan Xu,Jingyi Yu

DOI: https://doi.org/10.48550/arXiv.2104.11452

2021-07-16

Abstract:Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding in a data-driven multi-task manner. To enable robust capture under complex motion patterns, we propose an effective motion embedding module to recover both the implicit motion embedding and explicit 3D motion details via a corresponding mapping function as well as a sub-motion classifier. Based on such hybrid motion information, we introduce a multi-stream spatial-temporal Graph Convolutional Network(ST-GCN) to predict the fine-grained semantic action attributes, and adopt a semantic attribute mapping block to assemble various correlated action attributes into a high-level action label for the overall detailed understanding of the whole sequence, so as to enable various applications like action assessment or motion scoring. Comprehensive experiments on both public and our proposed datasets show that with a challenging monocular sports video input, our novel approach not only significantly improves the accuracy of 3D human motion capture, but also recovers accurate fine-grained semantic action attributes.

Computer Vision and Pattern Recognition

Neural monocular 3D human motion capture with physical awareness

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

MonoCap: Monocular Human Motion Capture Using a CNN Coupled with a Geometric Prior

NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action

MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency

Human Motion Tracking with Less Constraint of Initial Posture from a Single RGB-D Sensor

Real-time 3D neural facial animation from binocular video

Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video

Neural Capture of Animatable 3D Human from Monocular Video.

Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

Monocular 3D Human Pose Markerless Systems for Gait Assessment

Delving into Motion-Aware Matching for Monocular 3D Object Tracking

Decaf: Monocular Deformation Capture for Face and Hand Interactions

Accurate realtime full-body motion capture using a single depth camera

UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality

Towards robust 3d reconstruction of human motion from monocular video

Enhanced Spatio-Temporal Context for Temporally Consistent Robust 3D Human Motion Recovery from Monocular Videos

Real-time Physics-based Motion Capture with Sparse Sensors

4D Human Body Capture from Egocentric Video via 3D Scene Grounding

SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos