Abstract:This work targets at using a general deep learning framework to synthesize free-viewpoint images of arbitrary human performers, only requiring a sparse number of camera views as inputs and skirting per-case fine-tuning. The large variation of geometry and appearance, caused by articulated body poses, shapes and clothing types, are the key bottlenecks of this task. To overcome these challenges, we present a simple yet powerful framework, named Generalizable Neural Performer (GNR), that learns a generalizable and robust neural body representation over various geometry and appearance. Specifically, we compress the light fields for novel view human rendering as conditional implicit neural radiance fields from both geometry and appearance aspects. We first introduce an Implicit Geometric Body Embedding strategy to enhance the robustness based on both parametric 3D human body model and multi-view images hints. We further propose a Screen-Space Occlusion-Aware Appearance Blending technique to preserve the high-quality appearance, through interpolating source view appearance to the radiance fields with a relax but approximate geometric guidance. To evaluate our method, we present our ongoing effort of constructing a dataset with remarkable complexity and diversity. The dataset GeneBody-1.0, includes over 360M frames of 370 subjects under multi-view cameras capturing, performing a large variety of pose actions, along with diverse body shapes, clothing, accessories and hairdos. Experiments on GeneBody-1.0 and ZJU-Mocap show better robustness of our methods than recent state-of-the-art generalizable methods among all cross-dataset, unseen subjects and unseen poses settings. We also demonstrate the competitiveness of our model compared with cutting-edge case-specific ones. Dataset, code and model will be made publicly available.

Ivs-Net: Learning Human View Synthesis from Internet Videos

Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans

Deep Video‐Based Performance Synthesis from Sparse Multi‐View Capture

Implicit Neural Representations With Structured Latent Codes for Human Body Modeling

Novel View Synthesis of Humans using Differentiable Rendering

Novel View Synthesis of Dynamic Human with Sparse Cameras.

Novel View Synthesis of Human Interactions from Sparse Multi-view Videos

Novel View Synthesis from only a 6-DoF Camera Pose by Two-stage Networks

Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild

Human Pose Manipulation and Novel View Synthesis using Differentiable Rendering

Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition

Learning Dynamic Textures for Neural Rendering of Human Actors

Pseudo-Generalized Dynamic View Synthesis from a Video

Intrinsic Temporal Regularization for High-resolution Human Video Synthesis

AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation

Neural Rendering and Reenactment of Human Actor Videos

Neural Capture of Animatable 3D Human from Monocular Video.

Generating 3D-Consistent Videos from Unposed Internet Photos

Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors.

Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis

EVA-Gaussian: 3D Gaussian-based Real-time Human Novel View Synthesis under Diverse Camera Settings