Subjective and Objective Quality-of-Experience Evaluation Study for Live Video Streaming

Zehao Zhu,Wei Sun,Jun Jia,Wei Wu,Sibin Deng,Kai Li,Ying Chen,Xiongkuo Min,Jia Wang,Guangtao Zhai

2024-09-26

Abstract:In recent years, live video streaming has gained widespread popularity across various social media platforms. Quality of experience (QoE), which reflects end-users' satisfaction and overall experience, plays a critical role for media service providers to optimize large-scale live compression and transmission strategies to achieve perceptually optimal rate-distortion trade-off. Although many QoE metrics for video-on-demand (VoD) have been proposed, there remain significant challenges in developing QoE metrics for live video streaming. To bridge this gap, we conduct a comprehensive study of subjective and objective QoE evaluations for live video streaming. For the subjective QoE study, we introduce the first live video streaming QoE dataset, TaoLive QoE, which consists of $42$ source videos collected from real live broadcasts and $1,155$ corresponding distorted ones degraded due to a variety of streaming distortions, including conventional streaming distortions such as compression, stalling, as well as live streaming-specific distortions like frame skipping, variable frame rate, etc. Subsequently, a human study was conducted to derive subjective QoE scores of videos in the TaoLive QoE dataset. For the objective QoE study, we benchmark existing QoE models on the TaoLive QoE dataset as well as publicly available QoE datasets for VoD scenarios, highlighting that current models struggle to accurately assess video QoE, particularly for live content. Hence, we propose an end-to-end QoE evaluation model, Tao-QoE, which integrates multi-scale semantic features and optical flow-based motion features to predicting a retrospective QoE score, eliminating reliance on statistical quality of service (QoS) features.

Multimedia,Artificial Intelligence,Image and Video Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively evaluate the user's Quality of Experience (QoE) in real - time video streaming. Specifically, although there are already many QoE evaluation methods for Video on Demand (VoD), these methods pose significant challenges when applied to real - time video streaming. The main problems include:

1. **Lack of public real - time video databases**: Most of the existing QoE databases are based on the Video on Demand scenario and cannot accurately reflect phenomena such as frame rate changes and frame losses caused by network fluctuations in real - time video streaming.
2. **Imperfection of existing QoE model mechanisms**: Existing QoE models such as KSQI and GCNN - QoE rely on statistical features (for example, stuttering time, position, bit rate), which are difficult to obtain in advance in practical applications and are therefore not suitable for real - time live - streaming scenarios.

To address these problems, the paper proposes the following solutions:

- **Construct a large - scale real - time video database**: The paper creates a database named TaoLive QoE, which contains 42 high - quality source videos and 1,155 videos processed with different distortions. These videos cover multiple common real - time video quality problems such as compression distortion, stuttering events, accelerated playback, and frame losses.
- **Conduct subjective and objective QoE evaluations**: 23,100 subjective scores were collected through subjective experiments to generate the QoE scores of videos. At the same time, benchmark tests were carried out on existing QoE models and it was found that they perform poorly in real - time video scenarios.
- **Propose a new QoE evaluation model Tao - QoE**: This model is based on deep learning and can directly predict QoE scores from video inputs without relying on additional statistical information. The Tao - QoE model combines multi - scale semantic features and optical - flow motion features and can evaluate the QoE of videos more accurately.

Through these methods, the paper aims to fill the gap in the field of QoE evaluation for real - time video streaming and provide a more effective and practical QoE evaluation tool.