Abstract:Prediction of head movements in immersive media is key to designing efficient streaming systems able to focus the bandwidth budget on visible areas of the content. However, most of the numerous proposals made to predict user head motion in 360° images and videos do not explicitly consider a prominent characteristic of the head motion data: its intrinsic uncertainty. In this article, we present an approach to generate multiple plausible futures of head motion in 360° videos, given a common past trajectory. To our knowledge, this is the first work that considers the problem of multiple head motion prediction for 360° video streaming. We introduce our discrete variational multiple sequence (DVMS) learning framework, which builds on deep latent variable models. We design a training procedure to obtain a flexible, lightweight stochastic prediction model compatible with sequence-to-sequence neural architectures. Experimental results on 4 different datasets show that our method DVMS outperforms competitors adapted from the self-driving domain by up to 41% on prediction horizons up to 5 sec., at lower computational and memory costs. To understand how the learned features account for the motion uncertainty, we analyze the structure of the learned latent space and connect it with the physical properties of the trajectories. We also introduce a method to estimate the likelihood of each generated trajectory, enabling the integration of DVMS in a streaming system. We hence deploy an extensive evaluation of the interest of our DVMS proposal for a streaming system. To do so, we first introduce a new Python-based 360° streaming simulator that we make available to the community. On real-world user, video, and networking data, we show that predicting multiple trajectories yields higher fairness between the traces, the gains for 20 to 30% of the users reaching up to 10% in visual quality for the best number K of trajectories to generate.

VERSE: Virtual-Gradient Aware Streaming Lifelong Learning with Anytime Inference

Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis

Lifelong Learning of Video Diffusion Models From a Single Video Stream

Streaming View Learning.

Streaming Active Learning with Deep Neural Networks

A Unified Replay-based Continuous Learning Framework for Spatio-Temporal Prediction on Streaming Data

Learning with Feature Evolvable Streams.

AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning

Explainable Lifelong Stream Learning Based on "Glocal" Pairwise Fusion

A Simple Lifelong Learning Approach

Learning latent representations across multiple data domains using Lifelong VAEGAN

Memory Efficient Experience Replay for Streaming Learning

Deep Variational Learning for 360° Adaptive Streaming

Summarizing Stream Data for Memory-Constrained Online Continual Learning

Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams

Streaming Graph Neural Networks Via Continual Learning

Just a Glimpse: Rethinking Temporal Information for Video Continual Learning

Streaming Continual Learning for Unified Adaptive Intelligence in Dynamic Environments

Adaptive Streaming Continuous Learning System for Video Analytics

Adaptive online continual multi-view learning

Continual Learning Through Retrieval and Imagination.