Shot Boundary Detection Using Color Clustering and Attention Mechanism
Ye Yuan,Jiawan Zhang
DOI: https://doi.org/10.1145/3595923
2023-05-03
Abstract:Shot boundary detection (SBD) is widely used in scene segmentation, semantic analysis, and video retrieval. However, existing SBD algorithms have certain applications in video processing, but they have the following three problems. Firstly, these algorithms cannot effectively handle shot boundaries caused by sudden lighting changes. Secondly, when there are dimly lighting frames in the video, these algorithms cannot perform boundary detection well. Thirdly, when there is object or camera motion in the video, these algorithms also fail to work. To resolve these issues, we propose an SBD algorithm with color clustering changes in small regions (CCSR) to detect the shot transitions, which are abrupt changes and gradual transitions (dissolve and fade). The main idea behind the CCSR algorithm is to compute the distance of color features and to preserve the spatio-temporal information as much as possible. This model has relatively less dependence on the threshold parameters and sliding windows. Unlike other SBD algorithms, the clustering results of CCSR weaken factors such as object motion and illumination changes between adjacent frames in the video, which is helpful for reducing false detections. Furthermore, we utilize an attention mechanism in the gradual transitions to improve detection efficiency and accuracy. Finally, we evaluated the SBD algorithm, which was tested on a standard TRECVID dataset. The experimental results suggest that our algorithm yields significant improvements in precision and recall compared to the current techniques, with an average improvement of 10.35% and 8.85%, respectively. Moreover, compared with state-of-the-art algorithms, the results prove that the proposed method improves the F-score by more than 2.64% and the computation time efficiency by over 10%.
computer science, information systems, theory & methods, software engineering