Abstract:We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit space-time coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

Learning Dynamic Textures for Neural Rendering of Human Actors

Ivs-Net: Learning Human View Synthesis from Internet Videos

Neural Rendering and Reenactment of Human Actor Videos

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Learning Neural Volumetric Representations of Dynamic Humans in Minutes.

Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors.

Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures

Video-Driven Neural Physically-Based Facial Asset for Production

Real-time Deep Dynamic Characters

HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos

High-Fidelity Neural Human Motion Transfer from Monocular Video

Learning Compositional Radiance Fields of Dynamic Human Heads

Learning a Dynamic Neural Human Via Poses Guided Dual Spaces Feature

Efficient Integration of Neural Representations for Dynamic Humans

HDhuman: High-quality Human Novel-view Rendering from Sparse Views

Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans

HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs

Novel View Synthesis of Dynamic Human with Sparse Cameras.

Neural Capture of Animatable 3D Human from Monocular Video.

High-Fidelity Dynamic Human Synthesis via UV-Guided NeRF with Sparse Views

Relightable and Animatable Neural Avatar from Sparse-View Video