Abstract:Generalizable neural radiance field (NeRF) enables neural-based digital human rendering without per-scene retraining. When combined with human prior knowledge, high-quality human rendering can be achieved even with sparse input views. However, the inference of these methods is still slow, as a large number of neural network queries on each ray are required to ensure the rendering quality. Moreover, occluded regions often suffer from artifacts, especially when the input views are sparse. To address these issues, we propose a generalizable human NeRF framework that achieves high-quality and real-time rendering with sparse input views by extensively leveraging human prior knowledge. We accelerate the rendering with a two-stage sampling reduction strategy: first constructing boundary meshes around the human geometry to reduce the number of ray samples for sampling guidance regression, and then volume rendering using fewer guided samples. To improve rendering quality, especially in occluded regions, we propose an occlusion-aware attention mechanism to extract occlusion information from the human priors, followed by an image space refinement network to improve rendering quality. Furthermore, for volume rendering, we adopt a signed ray distance function (SRDF) formulation, which allows us to propose an SRDF loss at every sample position to improve the rendering quality further. Our experiments demonstrate that our method outperforms the state-of-the-art methods in rendering quality and has a competitive rendering speed compared with speed-prioritized novel view synthesis methods.

FloRen: Real-time High-quality Human Performance Rendering Via Appearance Flow Using Sparse RGB Cameras.

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

SAILOR: Synergizing Radiance and Occupancy Fields for Live Human Performance Capture

R2Human: Real-Time 3D Human Appearance Rendering from a Single Image

Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras

Learning Compositional Radiance Fields of Dynamic Human Heads

HDhuman: High-quality Human Performance Capture with Sparse Views

HDhuman: High-quality Human Novel-view Rendering from Sparse Views

FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from Sparse Views

High-Fidelity Human Avatars from a Single RGB Camera

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Neural Radiance Flow for 4D View Synthesis and Video Processing

EG-HumanNeRF: Efficient Generalizable Human NeRF Utilizing Human Prior for Sparse View

3D real-time human reconstruction with a single RGBD camera

PaReNeRF: Toward Fast Large-scale Dynamic NeRF with Patch-based Reference

HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video

FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces

Reconstructing Personalized Semantic Facial NeRF Models from Monocular Video

4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions

Geometry-guided generalizable NeRF for human rendering

Depth Guided Fast Rendering Neural Radiance Fields