ShadowMover: Automatically Projecting Real Shadows onto Virtual Object
Piaopiao Yu,Jie Guo,Fan Huang,Zhenyu Chen,Chen Wang,Yan Zhang,Yanwen Guo
DOI: https://doi.org/10.1109/tvcg.2023.3247066
IF: 5.2
2023-04-01
IEEE Transactions on Visualization and Computer Graphics
Abstract:Inserting 3D virtual objects into real-world images has many applications in photo editing and augmented reality. One key issue to ensure the reality of the composite whole scene is to generate consistent shadows between virtual and real objects. However, it is challenging to synthesize visually realistic shadows for virtual and real objects without any explicit geometric information of the real scene or manual intervention, especially for the shadows on the virtual objects projected by real objects. In view of this challenge, we present, to our knowledge, the first end-to-end solution to fully automatically project real shadows onto virtual objects for outdoor scenes. In our method, we introduce the Shifted Shadow Map, a new shadow representation that encodes the binary mask of shifted real shadows after inserting virtual objects in an image. Based on the shifted shadow map, we propose a CNN-based shadow generation model named ShadowMover which first predicts the shifted shadow map for an input image and then automatically generates plausible shadows on any inserted virtual object. A large-scale dataset is constructed to train the model. Our ShadowMover is robust to various scene configurations without relying on any geometric information of the real scene and is free of manual intervention. Extensive experiments validate the effectiveness of our method.
computer science, software engineering