Detecting and Removing Visual Distractors for Video Aesthetic Enhancement

Fang-Lue Zhang,Xian Wu,Rui-Long Li,Jue Wang,Zhao-Heng Zheng,Shi-Min Hu

DOI: https://doi.org/10.1109/tmm.2018.2790163

IF: 7.3

2018-01-01

IEEE Transactions on Multimedia

Abstract:Personal videos often contain visual distractors, which are objects that are accidentally captured and can distract viewers from focusing on the main subjects. We propose a method to automatically detect and localize these distractors through learning from a manually labeled dataset. To achieve spatially and temporally coherent detection, we propose extracting features at the temporal-superpixel level using a traditional supporting vector machine based learning framework. We also experiment with end-to-end learning using convolutional neural networks, which achieves slightly higher performance than other methods. The classification result is further refined in a postprocessing step based on graph-cut optimization. Experimental results show that our method achieves an accuracy of 81% and a recall of 86%. We demonstrate several ways of removing the detected distractors to improve the video quality, including video hole filling, video frame replacement, and camera path replanning. The user study results show that our method can significantly improve the aesthetic quality of videos.