SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage

Yifan Wang,Aleksander Holynski,Xiuming Zhang,Xuaner Zhang

DOI: https://doi.org/10.48550/arXiv.2204.03648

2022-04-07

Computer Vision and Pattern Recognition

Abstract:A light stage uses a series of calibrated cameras and lights to capture a subject's facial appearance under varying illumination and viewpoint. This captured information is crucial for facial reconstruction and relighting. Unfortunately, light stages are often inaccessible: they are expensive and require significant technical expertise for construction and operation. In this paper, we present SunStage: a lightweight alternative to a light stage that captures comparable data using only a smartphone camera and the sun. Our method only requires the user to capture a selfie video outdoors, rotating in place, and uses the varying angles between the sun and the face as guidance in joint reconstruction of facial geometry, reflectance, camera pose, and lighting parameters. Despite the in-the-wild un-calibrated setting, our approach is able to reconstruct detailed facial appearance and geometry, enabling compelling effects such as relighting, novel view synthesis, and reflectance editing. Results and interactive demos are available at https://sunstage.cs.washington.edu/.