Abstract:Synthesizing high-fidelity head avatars is a central problem for computer vision and graphics. While head avatar synthesis algorithms have advanced rapidly, the best ones still face great obstacles in real-world scenarios. One of the vital causes is inadequate datasets -- 1) current public datasets can only support researchers to explore high-fidelity head avatars in one or two task directions; 2) these datasets usually contain digital head assets with limited data volume, and narrow distribution over different attributes. In this paper, we present RenderMe-360, a comprehensive 4D human head dataset to drive advance in head avatar research. It contains massive data assets, with 243+ million complete head frames, and over 800k video sequences from 500 different identities captured by synchronized multi-view cameras at 30 FPS. It is a large-scale digital library for head avatars with three key attributes: 1) High Fidelity: all subjects are captured by 60 synchronized, high-resolution 2K cameras in 360 degrees. 2) High Diversity: The collected subjects vary from different ages, eras, ethnicities, and cultures, providing abundant materials with distinctive styles in appearance and geometry. Moreover, each subject is asked to perform various motions, such as expressions and head rotations, which further extend the richness of assets. 3) Rich Annotations: we provide annotations with different granularities: cameras' parameters, matting, scan, 2D/3D facial landmarks, FLAME fitting, and text description. Based on the dataset, we build a comprehensive benchmark for head avatar research, with 16 state-of-the-art methods performed on five main tasks: novel view synthesis, novel expression synthesis, hair rendering, hair editing, and talking head generation. Our experiments uncover the strengths and weaknesses of current methods. RenderMe-360 opens the door for future exploration in head avatars.

EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars

EMOCA: Emotion Driven Monocular Face Capture and Animation

MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devices

Audio-Driven Emotional Video Portraits

Emotional Speech-Driven Animation with Content-Emotion Disentanglement

EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

Automatic User-Specific Avatar Parametrisation and Emotion Mapping

EmoFace: Audio-driven Emotional 3D Face Animation

Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars

ExpAvatar: High-Fidelity Avatar Generation of Unseen Expressions with 3D Face Priors

Towards Rich Emotions in 3D Avatars: A Text-to-3D Avatar Generation Benchmark

Facial Expression Retargeting from Human to Avatar Made Easy

To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations

Expressive Gaussian Human Avatars from Monocular RGB Video

Neural Head Avatars from Monocular RGB Videos

ECAvatar: 3D Avatar Facial Animation with Controllable Identity and Emotion

Real-time Conversion from a Single 2D Face Image to a 3D Text-Driven Emotive Audio-Visual Avatar

One2Avatar: Generative Implicit Head Avatar For Few-shot User Adaptation

Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait

Can we truly transfer an actor's genuine happiness to avatars? An investigation into virtual, real, posed and spontaneous faces