Abstract:This paper introduces Motion-oriented Compositional Neural Radiance Fields (MoCo-NeRF), a framework designed to perform free-viewpoint rendering of monocular human videos via novel non-rigid motion modeling approach. In the context of dynamic clothed humans, complex cloth dynamics generate non-rigid motions that are intrinsically distinct from skeletal articulations and critically important for the rendering quality. The conventional approach models non-rigid motions as spatial (3D) deviations in addition to skeletal transformations. However, it is either time-consuming or challenging to achieve optimal quality due to its high learning complexity without a direct supervision. To target this problem, we propose a novel approach of modeling non-rigid motions as radiance residual fields to benefit from more direct color supervision in the rendering and utilize the rigid radiance fields as a prior to reduce the complexity of the learning process. Our approach utilizes a single multiresolution hash encoding (MHE) to concurrently learn the canonical T-pose representation from rigid skeletal motions and the radiance residual field for non-rigid motions. Additionally, to further improve both training efficiency and usability, we extend MoCo-NeRF to support simultaneous training of multiple subjects within a single framework, thanks to our effective design for modeling non-rigid motions. This scalability is achieved through the integration of a global MHE and learnable identity codes in addition to multiple local MHEs. We present extensive results on ZJU-MoCap and MonoCap, clearly demonstrating state-of-the-art performance in both single- and multi-subject settings. The code and model will be made publicly available at the project page: <a class="link-external link-https" href="https://stevejaehyeok.github.io/publications/moco-nerf" rel="external noopener nofollow">this https URL</a>.

Blender-NeRF: A Monocular Dynamic Human Body Explicit Reconstruction and Rendering Method

MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images

Neural Capture of Animatable 3D Human from Monocular Video.

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Animating NeRFs from Texture Space: A Framework for Pose-Dependent Rendering of Human Performances

AirNeRF: 3D Reconstruction of Human with Drone and NeRF for Future Communication Systems

Reconstructing Personalized Semantic Facial NeRF Models from Monocular Video

Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

Generalizable Neural Voxels for Fast Human Radiance Fields

MonoHuman: Animatable Human Neural Field from Monocular Video

MA-NeRF: Motion-Assisted Neural Radiance Fields for Face Synthesis from Sparse Images

HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields

4D Facial Avatar Reconstruction From Monocular Video via Efficient and Controllable Neural Radiance Fields

Efficient Neural Implicit Representation for 3D Human Reconstruction

High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

Dual-Space NeRF: Learning Animatable Avatars and Scene Lighting in Separate Spaces

HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos

Pre-NeRF 360: Enriching Unbounded Appearances for Neural Radiance Fields

Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos

Motion-Oriented Compositional Neural Radiance Fields for Monocular Dynamic Human Modeling

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields