Abstract:In this paper, we aim to address the challenge of novel view rendering of human performers that wear clothes with complex texture patterns using a sparse set of camera views. Although some recent works have achieved remarkable rendering quality on humans with relatively uniform textures using sparse views, the rendering quality remains limited when dealing with complex texture patterns as they are unable to recover the high-frequency geometry details that are observed in the input views. To this end, we propose HDhuman, which uses a human reconstruction network with a pixel-aligned spatial transformer and a rendering network with geometry-guided pixel-wise feature integration to achieve high-quality human reconstruction and rendering. The designed pixel-aligned spatial transformer calculates the correlations between the input views and generates human reconstruction results with high-frequency details. Based on the surface reconstruction results, the geometry-guided pixel-wise visibility reasoning provides guidance for multi-view feature integration, enabling the rendering network to render high-quality images at 2k resolution on novel views. Unlike previous neural rendering works that always need to train or fine-tune an independent network for a different scene, our method is a general framework that is able to generalize to novel subjects. Experiments show that our approach outperforms all the prior generic or specific methods on both synthetic data and real-world data. Source code and test data will be made publicly available for research purposes.

Novel View Synthesis of Humans using Differentiable Rendering

Human Pose Manipulation and Novel View Synthesis using Differentiable Rendering

Ivs-Net: Learning Human View Synthesis from Internet Videos

MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images

Novel View Synthesis of Dynamic Human with Sparse Cameras.

Novel-View Human Action Synthesis

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

HDhuman: High-quality Human Novel-view Rendering from Sparse Views

Novel View Synthesis of Human Interactions from Sparse Multi-view Videos

EVA-Gaussian: 3D Gaussian-based Real-time Human Novel View Synthesis under Diverse Camera Settings

Human Gaussian Splatting: Real-time Rendering of Animatable Avatars

Novel View Synthesis from only a 6-DoF Camera Pose by Two-stage Networks

Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras

Generative Novel View Synthesis with 3D-Aware Diffusion Models

Neural Rendering and Reenactment of Human Actor Videos

Novel View Synthesis from Single Images via Point Cloud Transformation

Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures

ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models

Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement