Abstract:Synthesizing high-fidelity head avatars is a central problem for computer vision and graphics. While head avatar synthesis algorithms have advanced rapidly, the best ones still face great obstacles in real-world scenarios. One of the vital causes is inadequate datasets -- 1) current public datasets can only support researchers to explore high-fidelity head avatars in one or two task directions; 2) these datasets usually contain digital head assets with limited data volume, and narrow distribution over different attributes. In this paper, we present RenderMe-360, a comprehensive 4D human head dataset to drive advance in head avatar research. It contains massive data assets, with 243+ million complete head frames, and over 800k video sequences from 500 different identities captured by synchronized multi-view cameras at 30 FPS. It is a large-scale digital library for head avatars with three key attributes: 1) High Fidelity: all subjects are captured by 60 synchronized, high-resolution 2K cameras in 360 degrees. 2) High Diversity: The collected subjects vary from different ages, eras, ethnicities, and cultures, providing abundant materials with distinctive styles in appearance and geometry. Moreover, each subject is asked to perform various motions, such as expressions and head rotations, which further extend the richness of assets. 3) Rich Annotations: we provide annotations with different granularities: cameras' parameters, matting, scan, 2D/3D facial landmarks, FLAME fitting, and text description. Based on the dataset, we build a comprehensive benchmark for head avatar research, with 16 state-of-the-art methods performed on five main tasks: novel view synthesis, novel expression synthesis, hair rendering, hair editing, and talking head generation. Our experiments uncover the strengths and weaknesses of current methods. RenderMe-360 opens the door for future exploration in head avatars.

ILSH: The Imperial Light-Stage Head Dataset for Human Head View Synthesis

Ivs-Net: Learning Human View Synthesis from Internet Videos

VSCHH 2023: A Benchmark for the View Synthesis Challenge of Human Heads

C3I-SynFace: A synthetic head pose and facial depth dataset using seed virtual human models.

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars

Head360: Learning a Parametric 3D Full-Head for Free-View Synthesis in 360°

Relightable Neural Human Assets from Multi-view Gradient Illuminations

VGGHeads: 3D Multi Head Alignment with a Large-Scale Synthetic Dataset

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Lite2Relight: 3D-aware Single Image Portrait Relighting

Objects With Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relighting

Multiface: A Dataset for Neural Face Rendering

HSPACE: Synthetic Parametric Humans Animated in Complex Environments

LUCES: A Dataset for Near-Field Point Light Source Photometric Stereo

PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$

SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage

SynthOutdoor: A synthetic dataset for 3D outdoor light estimation

NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting

HDhuman: High-quality Human Novel-view Rendering from Sparse Views

Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures

SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis