MVBIND: Self-Supervised Music Recommendation For Videos Via Embedding Space Binding

Jiajie Teng,Huiyu Duan,Yucheng Zhu,Sijing Wu,Guangtao Zhai

2024-05-15

Abstract:Recent years have witnessed the rapid development of short videos, which usually contain both visual and audio modalities. Background music is important to the short videos, which can significantly influence the emotions of the viewers. However, at present, the background music of short videos is generally chosen by the video producer, and there is a lack of automatic music recommendation methods for short videos. This paper introduces MVBind, an innovative Music-Video embedding space Binding model for cross-modal retrieval. MVBind operates as a self-supervised approach, acquiring inherent knowledge of intermodal relationships directly from data, without the need of manual annotations. Additionally, to compensate the lack of a corresponding musical-visual pair dataset for short videos, we construct a dataset, SVM-10K(Short Video with Music-10K), which mainly consists of meticulously selected short videos. On this dataset, MVBind manifests significantly improved performance compared to other baseline methods. The constructed dataset and code will be released to facilitate future research.

Multimedia,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the automatic recommendation of short - sighted background music. Specifically, with the rapid development of short - videos, background music plays an important role in the emotional expression of short - videos, as well as the understanding and experience of the audience. However, currently, the background music of short - videos is mainly selected by video producers, lacking effective automatic music recommendation methods. This not only consumes a great deal of time and energy of video producers, but also makes it difficult to efficiently select appropriate background music for different video clips when facing a large amount of music and video data. Therefore, it becomes particularly important to develop a music recommendation system that can improve the production efficiency of short - videos.

The paper proposes a self - supervised Music - Video embedding space Binding model named MVBind, aiming to achieve cross - modal retrieval between music and video modalities. MVBind directly obtains the relational knowledge between modalities from data in a self - supervised manner without manual annotation. In addition, in order to make up for the lack of corresponding music - visual paired datasets, the researchers constructed a dataset SVM - 10K containing nearly 10,000 carefully selected short - videos. The experimental results show that on the SVM - 10K dataset, MVBind shows a significant performance improvement compared with other baseline methods.