Abstract:We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit space-time coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

Animation Of 3d Characters From Single Depth Camera

Sketch Based Multi-source 3D Animation Transfer

Animating Scanned Human Models.

Interactive Animation Generation of Virtual Characters Using Single RGB-D Camera

Markerless Body Motion Capturing for 3D Character Animation based on Multi-view Cameras

Real-time Animation and Motion Retargeting of Virtual Characters Based on Single RGB-D Camera.

3D Articulated Skeleton Extraction Using a Single Consumer-Grade Depth Camera.

Performance Capture Of Interacting Characters With Handheld Kinect

Human Performance Capture Using Multiple Handheld Kinects

Scanning 3D Full Human Bodies Using Kinects

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Human Motion Capture Using Wireless Inertial Sensors

Accurate realtime full-body motion capture using a single depth camera

Video Based Human Animation Technique

Sketch-based skeleton-driven 2D animation and motion capture

Video-based Characters

Animation of 3D Human Model Using Markerless Motion Capture Applied To Sports

A Realization Method of 3D Virtual Human Animation Driven By Video

Bodyfusion: Real-Time Capture Of Human Motion And Surface Geometry Using A Single Depth Camera

Automatic Conversion of Human Mesh into Skeleton Animation by Using Kinect Motion

Real-time Deep Dynamic Characters