Abstract:The diversity of video delivery pipeline poses a grand challenge to the evaluation of adaptive bitrate (ABR) streaming algorithms and objective quality-of-experience (QoE) models. Here we introduce so-far the largest subject-rated database of its kind, namely WaterlooSQoE-IV, consisting of 1350 adaptive streaming videos created from diverse source contents, video encoders, network traces, ABR algorithms, and viewing devices. We collect human opinions for each video with a series of carefully designed subjective experiments. Subsequent data analysis and testing/comparison of ABR algorithms and QoE models using the database lead to a series of novel observations and interesting findings, in terms of the effectiveness of subjective experiment methodologies, the interactions between user experience and source content, viewing device and encoder type, the heterogeneities in the bias and preference of user experiences, the behaviors of ABR algorithms, and the performance of objective QoE models. Most importantly, our results suggest that a better objective QoE model, or a better understanding of human perceptual experience and behaviour, is the most dominating factor in improving the performance of ABR algorithms, as opposed to advanced optimization frameworks, machine learning strategies or bandwidth predictors, where a majority of ABR research has been focused on in the past decade. On the other hand, our performance evaluation of 11 QoE models shows only a moderate correlation between state-of-the-art QoE models and subjective ratings, implying rooms for improvement in both QoE modeling and ABR algorithms. The database is made publicly available at: \url{<a class="link-external link-https" href="https://ece.uwaterloo.ca/~zduanmu/waterloosqoe4/" rel="external noopener nofollow">this https URL</a>}.

Bitstream-based quality assessment for networked video: a review

From QoS to QoE: A Tutorial on Video Quality Assessment.

Encoder Network Decoder Original video Distorted video Coding & Compression Distortions Transmission Network Distortions End user External Factors Fig . 2 : Video transmission path videos usually exhibit the following typical visual distortions

Evaluation of No Reference Bitstream-based Video Quality Assessment Methods.

Quality Assessment and Measurement for Internet Video Streaming

Interactive H.264/SVC Error Control Methods Design and Implementation

Modeling the Impact of Frame Rate and Quantization Stepsizes and Their Temporal Variations on Perceptual Video Quality: A Review of Recent Works

A Method of Video Quality Assessment Based on the Sensitive Region.

Hybrid Distortion Ranking Tuned Bitstream-Layer Video Quality Assessment

Assessing quality of experience for adaptive HTTP video streaming

A Brief Survey on Adaptive Video Streaming Quality Assessment

Perceptual video quality assessment: a survey

Effective Quality Analysis for Video Streaming over Wireless Ad Hoc Network

Cross-Dimensional Perceptual Quality Assessment for Low Bit-Rate Videos

Cross-Dimensional Quality Assessment For Low Bitrate Video

A Content-Based Bit Allocation Model For Video Streaming

Assessing the Quality-of-Experience of Adaptive Bitrate Video Streaming

Model and Performance of a No-Reference Quality Assessment Metric for Video Streaming

Adaptive Video Streaming with Optimized Bitstream Extraction and PID-Based Quality Control

Video Quality Metric For Low Bitrate Compressed Videos