Abstract:Eyeglasses play an important role in the perception of identity. Authentic virtual representations of faces can benefit greatly from their inclusion. However, modeling the geometric and appearance interactions of glasses and the face of virtual representations of humans is challenging. Glasses and faces affect each other's geometry at their contact points, and also induce appearance changes due to light transport. Most existing approaches do not capture these physical interactions since they model eyeglasses and faces independently. Others attempt to resolve interactions as a 2D image synthesis problem and suffer from view and temporal inconsistencies. In this work, we propose a 3D compositional morphable model of eyeglasses that accurately incorporates high-fidelity geometric and photometric interaction effects. To support the large variation in eyeglass topology efficiently, we employ a hybrid representation that combines surface geometry and a volumetric representation. Unlike volumetric approaches, our model naturally retains correspondences across glasses, and hence explicit modification of geometry, such as lens insertion and frame deformation, is greatly simplified. In addition, our model is relightable under point lights and natural illumination, supporting high-fidelity rendering of various frame materials, including translucent plastic and metal within a single morphable model. Importantly, our approach models global light transport effects, such as casting shadows between faces and glasses. Our morphable model for eyeglasses can also be fit to novel glasses via inverse rendering. We compare our approach to state-of-the-art methods and demonstrate significant quality improvements.

Making 3D Eyeglasses Try-on Practical

Magic Glasses: From 2D to 3D.

Holographic near-eye display system based on double-convergence light Gerchberg-Saxton algorithm.

Mesh deformation-based single-view 3D reconstruction of thin eyeglasses frames with differentiable rendering

Real time stereo rendering for augmented reality on 3DTV system

Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

Designing an AI-Based Virtual Try-On Web Application

Development of a Visual Reality Headset and Its Applications in 3D Interactive Bike Race Games

Development of a Three-Dimensional Multimode Visual Immersive System With Applications in Telepresence

Real-Time 3D Eye Performance Reconstruction for RGBD Cameras

Retinal 3D

DEEP: 3D Gaze Pointing in Virtual Reality Leveraging Eyelid Movement

Realistic Monocular-To-3d Virtual Try-On Via Multi-Scale Characteristics Capture

Text-guided Eyeglasses Manipulation with Spatial Constraints

Facial performance sensing head-mounted display

3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction `

MEGANE: Morphable Eyeglass and Avatar Network

Three stages of 3D virtual try-on network with appearance flow and shape field

EasyGaze3D: Towards Effective and Flexible 3D Gaze Estimation from a Single RGB Camera

Low-Complexity 3D-Vision Conferencing System Based on Accelerated RIFE Model

3D Volume Visualization and Screen-based Interaction with Dynamic Ray Casting on Autostereoscopic Display