Abstract:This work targets at using a general deep learning framework to synthesize free-viewpoint images of arbitrary human performers, only requiring a sparse number of camera views as inputs and skirting per-case fine-tuning. The large variation of geometry and appearance, caused by articulated body poses, shapes and clothing types, are the key bottlenecks of this task. To overcome these challenges, we present a simple yet powerful framework, named Generalizable Neural Performer (GNR), that learns a generalizable and robust neural body representation over various geometry and appearance. Specifically, we compress the light fields for novel view human rendering as conditional implicit neural radiance fields from both geometry and appearance aspects. We first introduce an Implicit Geometric Body Embedding strategy to enhance the robustness based on both parametric 3D human body model and multi-view images hints. We further propose a Screen-Space Occlusion-Aware Appearance Blending technique to preserve the high-quality appearance, through interpolating source view appearance to the radiance fields with a relax but approximate geometric guidance. To evaluate our method, we present our ongoing effort of constructing a dataset with remarkable complexity and diversity. The dataset GeneBody-1.0, includes over 360M frames of 370 subjects under multi-view cameras capturing, performing a large variety of pose actions, along with diverse body shapes, clothing, accessories and hairdos. Experiments on GeneBody-1.0 and ZJU-Mocap show better robustness of our methods than recent state-of-the-art generalizable methods among all cross-dataset, unseen subjects and unseen poses settings. We also demonstrate the competitiveness of our model compared with cutting-edge case-specific ones. Dataset, code and model will be made publicly available.

Voxel Grid Performer: Efficient Radiance Fields Generalization for Human Novel View Synthesis

MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images

Generalizable Neural Voxels for Fast Human Radiance Fields

Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis

HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs

VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs

Neural Deformable Voxel Grid for Fast Optimization of Dynamic View Synthesis

GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images

Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering

SAILOR: Synergizing Radiance and Occupancy Fields for Live Human Performance Capture

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

HDhuman: High-quality Human Novel-view Rendering from Sparse Views

MVoxTi-DNeRF: Explicit Multi-Scale Voxel Interpolation and Temporal Encoding Network for Efficient Dynamic Neural Radiance Field

VoxNeRF: Bridging Voxel Representation and Neural Radiance Fields for Enhanced Indoor View Synthesis

EG-HumanNeRF: Efficient Generalizable Human NeRF Utilizing Human Prior for Sparse View

Geometry-guided generalizable NeRF for human rendering

Learning Neural Volumetric Representations of Dynamic Humans in Minutes.

Voxel-Mesh Hybrid Representation for Real-Time View Synthesis

High-Fidelity Dynamic Human Synthesis via UV-Guided NeRF with Sparse Views

HDhuman: High-quality Human Performance Capture with Sparse Views

VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids