Abstract:In this paper, we focus on the challenges of modeling deformable 3D objects from casual videos. With the popularity of neural radiance fields (NeRF), many works extend it to dynamic scenes with a canonical NeRF and a deformation model that achieves 3D point transformation between the observation space and the canonical space. Recent works rely on linear blend skinning (LBS) to achieve the canonical-observation transformation. However, the linearly weighted combination of rigid transformation matrices is not guaranteed to be rigid. As a matter of fact, unexpected scale and shear factors often appear. In practice, using LBS as the deformation model can always lead to skin-collapsing artifacts for bending or twisting motions. To solve this problem, we propose neural dual quaternion blend skinning (NeuDBS) to achieve 3D point deformation, which can perform rigid transformation without skin-collapsing artifacts. In the endeavor to register 2D pixels across different frames, we establish a correspondence between canonical feature embeddings that encodes 3D points within the canonical space, and 2D image features by solving an optimal transport problem. Besides, we introduce a texture filtering approach for texture rendering that effectively minimizes the impact of noisy colors outside target deformable objects. Extensive experiments on real and synthetic datasets show that our approach can reconstruct 3D models for humans and animals with better qualitative and quantitative performance than state-of-the-art methods. Project page: \url{<a class="link-external link-https" href="https://chaoyuesong.github.io/MoDA" rel="external noopener nofollow">this https URL</a>}.

Learning a Dynamic Neural Human Via Poses Guided Dual Spaces Feature

Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans

Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos

LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Learning Pose Controllable Human Reconstruction with Dynamic Implicit Fields from a Single Image

Learning Dynamic Textures for Neural Rendering of Human Actors

Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors.

Deep Dual Consecutive Network for Human Pose Estimation

Implicit Neural Representations With Structured Latent Codes for Human Body Modeling

MonoHuman: Animatable Human Neural Field from Monocular Video

Temporally Coherent Full 3D Mesh Human Pose Recovery from Monocular Video

3D Human pose estimation from video via multi-scale multi-level spatial temporal features

Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

Multimodal Deep Autoencoder for Human Pose Recovery

Relightable and Animatable Neural Avatar from Sparse-View Video

HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos

Deep3DPose: Realtime Reconstruction of Arbitrarily Posed Human Bodies from Single RGB Images

Neural Capture of Animatable 3D Human from Monocular Video.

Detailed Human Shape Estimation From A Single Image By Hierarchical Mesh Deformation

MoDA: Modeling Deformable 3D Objects from Casual Videos