A spherical mixture model approach for 360 video virtual cinematography

Chenglei Wu,Zhi Wang,Lifeng Sun

DOI: https://doi.org/10.1109/ICIP42928.2021.9506445

2021-01-01

Abstract:360 video virtual cinematography attempts to direct a virtual camera and capture the most salient regions of 360 videos. In this paper, we propose a data-drive solution to achieve high-quality and diversified 360 cinematography based on crowd-sourced viewing histories. Specifically, we try to address two problems: 1) how to locate the semantically important regions of interest (RoI) from raw data, 2) how to generate virtual camera paths that follow chronological narratives. We first design a dynamic spherical mixture model based algorithm to locate variable number of RoIs on each video frame. We then model the camera transition and chronological orders with a Bayesian network and conditional probabilities. With the above two designs, we can generate "optimal" cinematography paths based on a dynamic programming algorithm. By modeling the RoIs as spherical mixture model, we are also able to provide diversified cinematography results. We show its effectiveness through extensive experiments.