Inverting Generative Adversarial Renderer for Face Reconstruction

Jingtan Piao,Keqiang Sun,KwanYee Lin,Quan Wang,Hongsheng Li

DOI: https://doi.org/10.48550/arXiv.2105.02431

2021-05-08

Abstract:Given a monocular face image as input, 3D face geometry reconstruction aims to recover a corresponding 3D face mesh. Recently, both optimization-based and learning-based face reconstruction methods have taken advantage of the emerging differentiable renderer and shown promising results. However, the differentiable renderer, mainly based on graphics rules, simplifies the realistic mechanism of the illumination, reflection, \etc, of the real world, thus cannot produce realistic images. This brings a lot of domain-shift noise to the optimization or training process. In this work, we introduce a novel Generative Adversarial Renderer (GAR) and propose to tailor its inverted version to the general fitting pipeline, to tackle the above problem. Specifically, the carefully designed neural renderer takes a face normal map and a latent code representing other factors as inputs and renders a realistic face image. Since the GAR learns to model the complicated real-world image, instead of relying on the simplified graphics rules, it is capable of producing realistic images, which essentially inhibits the domain-shift noise in training and optimization. Equipped with the elaborated GAR, we further proposed a novel approach to predict 3D face parameters, in which we first obtain fine initial parameters via Renderer Inverting and then refine it with gradient-based optimizers. Extensive experiments have been conducted to demonstrate the effectiveness of the proposed generative adversarial renderer and the novel optimization-based face reconstruction framework. Our method achieves state-of-the-art performances on multiple face reconstruction datasets.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to accurately reconstruct the corresponding 3D facial geometric structure given a monocular face image. Existing optimization - based and learning - based methods encounter some challenges when using differentiable renderers, mainly because these renderers rely on simplified graphic rules and cannot generate highly realistic images, resulting in domain - shift noise during the training or optimization process. Moreover, these methods are difficult to optimize because they can only back - propagate errors to local vertices.

To overcome these problems, the paper proposes a novel Generative Adversarial Renderer (GAR) and customizes it into a general fitting pipeline through its inverse version to solve the above problems. Specifically, the designed neural renderer takes the face normal map and the latent code representing other factors as input and renders a realistic face image. Since GAR learns to model complex real - world images instead of relying on simplified graphic rules, it can generate realistic images and fundamentally suppress the domain - shift noise in training and optimization. By combining the carefully designed GAR, the paper further proposes a new method to predict 3D face parameters. First, fine initial parameters are obtained through the inverse of the renderer, and then refined using a gradient - based optimizer. A large number of experiments have verified the effectiveness of the proposed Generative Adversarial Renderer and the new optimization - based face reconstruction framework. This method has achieved state - of - the - art performance on multiple facial reconstruction datasets.