Abstract:Prediction of head movements in immersive media is key to designing efficient streaming systems able to focus the bandwidth budget on visible areas of the content. However, most of the numerous proposals made to predict user head motion in 360° images and videos do not explicitly consider a prominent characteristic of the head motion data: its intrinsic uncertainty. In this article, we present an approach to generate multiple plausible futures of head motion in 360° videos, given a common past trajectory. To our knowledge, this is the first work that considers the problem of multiple head motion prediction for 360° video streaming. We introduce our discrete variational multiple sequence (DVMS) learning framework, which builds on deep latent variable models. We design a training procedure to obtain a flexible, lightweight stochastic prediction model compatible with sequence-to-sequence neural architectures. Experimental results on 4 different datasets show that our method DVMS outperforms competitors adapted from the self-driving domain by up to 41% on prediction horizons up to 5 sec., at lower computational and memory costs. To understand how the learned features account for the motion uncertainty, we analyze the structure of the learned latent space and connect it with the physical properties of the trajectories. We also introduce a method to estimate the likelihood of each generated trajectory, enabling the integration of DVMS in a streaming system. We hence deploy an extensive evaluation of the interest of our DVMS proposal for a streaming system. To do so, we first introduce a new Python-based 360° streaming simulator that we make available to the community. On real-world user, video, and networking data, we show that predicting multiple trajectories yields higher fairness between the traces, the gains for 20 to 30% of the users reaching up to 10% in visual quality for the best number K of trajectories to generate.

Streaming variational bayes

Streaming Gibbs Sampling for LDA Model

Streaming Parallel Variational Bayesian Supervised Factor Analysis for Adaptive Soft Sensor Modeling with Big Process Data

Streaming Variational Monte Carlo

Streaming Bayes GFlowNets

Variational Bayesian Inference for Robust Streaming Tensor Factorization and Completion

A Novel Streaming Variational Bayesian Supervised Factor Analysis For Industrial Adaptive Soft Sensor Modeling

Variance Reduced Diffusion Adaptation for Online Learning over Networks

Robust Factorization and Completion of Streaming Tensor Data via Variational Bayesian Inference

A trust-region method for stochastic variational inference with applications to streaming data

Bayesian Stochastic Gradient Descent for Stochastic Optimization with Streaming Input Data

Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent

Deep Variational Learning for 360° Adaptive Streaming

Streaming Bayesian inference: theoretical limits and mini-batch approximate message-passing

Distributed Variational Bayesian Algorithms Over Sensor Networks

Distributed Bayesian Inference Over Sensor Networks

Auto-Encoding Variational Bayes

Memorized Variational Continual Learning for Dirichlet Process Mixtures

Scalable and adaptive variational Bayes methods for Hawkes processes

Learning Graphical Models from a Distributed Stream