Vision-Language Models pre-trained on large-scale image-text datasets have
shown superior performance in downstream tasks such as image retrieval. Most of
the images for pre-training are presented in the form of open domain
common-sense visual elements. Differently, video covers in short video search
scenarios are presented as user-originated contents that provide important
visual summaries of videos. In addition, a portion of the video covers come
with manually designed cover texts that provide semantic complements. In order
to fill in the gaps in short video cover data, we establish the first
large-scale cover-text benchmark for Chinese short video search scenarios.
Specifically, we release two large-scale datasets CBVS-5M/10M to provide short
video covers, and the manual fine-labeling dataset CBVS-20K to provide real
user queries, which serves as an image-text benchmark test in the Chinese short
video search field. To integrate the semantics of cover text in the case of
modality missing, we propose UniCLIP where cover texts play a guiding role
during training, however are not relied upon by inference. Extensive evaluation
on CBVS-20K demonstrates the excellent performance of our proposal. UniCLIP has
been deployed to Tencent's online video search systems with hundreds of
millions of visits and achieved significant gains. The complete dataset, code
and checkpoints will be available upon release.

利用大规模图像文本预训练的视觉语言模型在图像检索等任务中表现出优异性能；为了填补短视频封面数据的缺失，我们建立了首个面向中文短视频搜索场景的大规模封面文本基准测试集，并提出了 UniCLIP 方法，通过训练时的封面文本引导，但推理时不依赖封面文本来集成语义；在经过大规模验证数据集 CBVS-20K 的深度评估中，我们的提案 UniCLIP 表现出了出色的性能，并在腾讯在线视频搜索系统上部署，取得了显著的增益。