CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios

Xiangshuo Qiao,Xianxin Li,Xiaozhe Qu,Jie Zhang,Yang Liu,Yu Luo,Cihang Jin,Jin Ma

2024-01-25

Abstract:Vision-Language Models pre-trained on large-scale image-text datasets have shown superior performance in downstream tasks such as image retrieval. Most of the images for pre-training are presented in the form of open domain common-sense visual elements. Differently, video covers in short video search scenarios are presented as user-originated contents that provide important visual summaries of videos. In addition, a portion of the video covers come with manually designed cover texts that provide semantic complements. In order to fill in the gaps in short video cover data, we establish the first large-scale cover-text benchmark for Chinese short video search scenarios. Specifically, we release two large-scale datasets CBVS-5M/10M to provide short video covers, and the manual fine-labeling dataset CBVS-20K to provide real user queries, which serves as an image-text benchmark test in the Chinese short video search field. To integrate the semantics of cover text in the case of modality missing, we propose UniCLIP where cover texts play a guiding role during training, however are not relied upon by inference. Extensive evaluation on CBVS-20K demonstrates the excellent performance of our proposal. UniCLIP has been deployed to Tencent's online video search systems with hundreds of millions of visits and achieved significant gains. The dataset and code are available at

Computer Vision and Pattern Recognition,Multimedia

What problem does this paper attempt to address?

This paper introduces a large-scale image-text benchmark called CBVS (Chinese Image-Text Benchmark for Short Video Search) for the Chinese short video search scenario. Existing pre-trained visual language models are mostly based on open-domain image data, while the video cover images in short video search are user-generated content with different characteristics, such as manually composed visual elements and auxiliary cover text. To fill this data gap, the paper establishes the CBVS dataset, which includes CBVS-20K (manually annotated) and CBVS-5M/10M (large-scale unsupervised) parts, providing information such as short video covers, titles, and user queries.

CBVS-20K contains 20,000 high-quality <user query - video cover> pairs for evaluating image-text matching performance, while CBVS-5M/10M is used for pre-training or fine-tuning tasks and contains approximately 5 million/10 million <video title - video cover> pairs. In addition, the paper proposes the UniCLIP model, which guides image-text contrastive learning by introducing image classification tasks and image-text matching tasks, enabling effective integration of cover image semantics even in the absence of cover text modality.

Experiments show that UniCLIP performs excellently on CBVS-20K and has been deployed in Tencent's online video search system, bringing significant performance improvements. This work is an important contribution to Chinese multimodal representation learning and short video search, providing new datasets and model design ideas.