Robust Ultralow Bitrate Video Conferencing with Second Order Motion Coherency

Zhehao Chen,Ming Lu,Hao Chen,Zhan Ma

DOI: https://doi.org/10.1109/mmsp55362.2022.9949138

2022-01-01

Abstract:The emergence of unsupervised deep image animation (DIA) has promised unprecedented prospects of video conferencing applications across ultralow-bandwidth networks. Existing DIA approaches rely on the First Order Motion (FOM) Model to combine compressed sparse motion features (SMFs) from motion driving frames and the appearance feature extracted from the source image for high-quality video generation. This work improves the FOM model by introducing the Second Order Motion (SOM) Coherency for better synthesis, with which we not only best assure the temporal smoothness that is not considered in FOM, but also enable the effective compensation of packet loss often encountered in real-life scenarios. Extensive experiments show that our method outperforms the HEVC-based conferencing with $\approx \mathbf{70}\%$ BD-rate gains with network bandwidth $< \mathbf{10}$ kbps, and surpasses state-of-art solution proposed by Konuko et al. with about 30 absolute percentage points improvement in BD-rate.