Panoramic Image Generation: From 2-D Sketch to Spherical Image
Yiping Duan,Chaoyi Han,Xiaoming Tao,Bingrui Geng,Yunfei Du,Jianhua Lu
DOI: https://doi.org/10.1109/jstsp.2020.2968772
IF: 7.695
2020-01-01
IEEE Journal of Selected Topics in Signal Processing
Abstract:The 360-degree video/image, also called an omnidirectional video/image or panoramic video/image, is very important in some emerging areas such as virtual reality (VR). Therefore, corresponding image generation algorithms are urgently needed. However, existing image generation models mainly focus on 2-D images and do not consider the spherical structures of panoramic images. In this article, we propose a panoramic image generation method based on spherical convolution and generative adversarial networks, called spherical generative adversarial networks (SGANs). We adopt the sketch map as the input, which is a concise geometric structure representation of the panoramic image, e.g., comprising approximately 7% of the pixels for a 583 × 1163 image. Through adversarial learning, a realistic-looking, plausible and high-fidelity spherical image can be obtained from the sparse sketch map. In particular, we build a dataset of the sketch maps using a visual computation-based sketching model. Then, by optimizing SGANs with GAN loss, feature matching loss and perceptual loss, realistic textures and details are recovered gradually. On one hand, it is an improvement using the sparse sketch map as input rather than the denser input, e.g., the features of the textures and colors. On the other hand, spherical convolution helps to remedy space-varying distortions of the planar projection. We conduct extensive experiments on some public panoramic image datasets and compare them with state-of-the-art techniques to validate the superior performance of the proposed approach.