Abstract:As deep convolutional neural networks (DNNs) are widely used in various fields of computer vision, leveraging the overfitting ability of the DNN to achieve video resolution upscaling has become a new trend in the modern video delivery system. By dividing videos into chunks and overfitting each chunk with a super-resolution model, the server encodes videos before transmitting them to the clients, thus achieving better video quality and transmission efficiency. However, a large number of chunks are expected to ensure good overfitting quality, which substantially increases the storage and consumes more bandwidth resources for data transmission. On the other hand, decreasing the number of chunks through training optimization techniques usually requires high model capacity, which significantly slows down execution speed. To reconcile such, we propose a novel method for high-quality and efficient video resolution upscaling tasks, which leverages the spatial-temporal information to accurately divide video into chunks, thus keeping the number of chunks as well as the model size to minimum. Additionally, we advance our method into a single overfitting model by a data-aware joint training technique, which further reduces the storage requirement with negligible quality drop. We deploy our models on an off-the-shelf mobile phone, and experimental results show that our method achieves real-time video super-resolution with high video quality. Compared with the state-of-the-art, our method achieves 28 fps streaming speed with 41.6 PSNR, which is 14$\times$ faster and 2.29 dB better in the live video resolution upscaling tasks. Code available in <a class="link-external link-https" href="https://github.com/coulsonlee/STDO-CVPR2023.git" rel="external noopener nofollow">this https URL</a>

Neural Network-Based Ultra-High-Definition Video Live Streaming Optimization Algorithm

Video Super-Resolution Based on Multiple Networks Merging

Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting

Real-Time CNN Training and Compression for Neural-Enhanced Adaptive Live Streaming

Online Streaming Video Super-Resolution with Convolutional Look-Up Table

Super Efficient Neural Network for Compression Artifacts Reduction and Super Resolution

A Dual-Network Based Super-Resolution for Compressed High Definition Video.

Video Super-Resolution Algorithm Based on Spatial-Temporal Feature and Neural Network

Super Resolution Method for Network Video

AsConvSR: Fast and Lightweight Super-Resolution Network with Assembled Convolutions

Learning-Based Quality Enhancement for Scalable Coded Video over Packet Lossy Networks

Enhancement or Super-Resolution: Learning-based Adaptive Video Streaming with Client-Side Video Processing

Higher Quality Live Streaming under Lower Uplink Bandwidth

Prediction-assistant Frame Super-Resolution for Video Streaming

GPU Accelerated High-Quality Video/image Super-Resolution

Real-world video superresolution enhancement method based on the adaptive down-sampling model

An Efficient Network Design for Face Video Super-resolution

Perceptual Quality Enhancement with Multi-scale Deep Learning for Video Transmission: A QoE Perspective

Queue-Learning-Based QoE Optimization for Super-Resolution-Assisted Adaptive Video Streaming.

Robust Saliency-Driven Quality Adaptation for Mobile 360-Degree Video Streaming