Quality of Experience Evaluation Model with No-Reference VMAF Metric and Deep Spatio-temporal Features of Video
Du Lina,Yang Shuo,Zhuo Li,Zhang Hui,Zhang Jing,Li Jiafeng
DOI: https://doi.org/10.1007/s11220-022-00386-2
2022-01-01
Sensing and Imaging
Abstract:For mobile streaming media service providers, it is necessary to accurately predict the quality of experience (QoE) to formulate appropriate resource allocation and service quality optimization strategies. In this paper, a QoE evaluation model is proposed by considering various influencing factors (IFs), including perceptual video quality, video content characteristics, stalling, quality switching and video genre attribute. Firstly, a no-reference video multimethod assessment fusion (VMAF) model is constructed to measure the perceptual quality of the video by the deep bilinear convolutional neural network. Then, the deep spatio-temporal features of video are extracted using a TSM-ResNet50 network, which incorporates temporal shift module (TSM) with ResNet50, obtaining feature representation of video content characteristics while balancing computational efficiency and expressive ability. Secondly, video genre attribute, which reflects the user’s preference for different types of videos, is considered as a IF while constructing the QoE model. The statistical parameters of other IFs, including the video genre attribute, stalling and quality switching, are combined with VMAF and deep spatio-temporal features of video to form an overall description parameters vector of IFs for formulating the QoE evaluation model. Finally, the mapping relationship model between the parameters vector of IFs and the mean opinion score is established through designing a deep neural network. The proposed QoE evaluation model is validated on two public video datasets: WaterlooSQoE-III and LIVE-NFLX-II. The experimental results show that the proposed model can achieve the state-of-the-art QoE prediction performance.