Current metrics for video captioning are mostly based on the text-level
comparison between reference and candidate captions. However, they have some
insuperable drawbacks, e.g., they cannot handle videos without references, and
they may result in biased evaluation due to the one-to-many nature of
video-to-text and the neglect of visual relevance. From the human evaluator's
viewpoint, a high-quality caption should be consistent with the provided video,
but not necessarily be similar to the reference in literal or semantics.
Inspired by human evaluation, we propose EMScore (Embedding Matching-based
score), a novel reference-free metric for video captioning, which directly
measures similarity between video and candidate captions. Benefit from the
recent development of large-scale pre-training models, we exploit a well
pre-trained vision-language model to extract visual and linguistic embeddings
for computing EMScore. Specifically, EMScore combines matching scores of both
coarse-grained (video and caption) and fine-grained (frames and words) levels,
which takes the overall understanding and detailed characteristics of the video
into account. Furthermore, considering the potential information gain, EMScore
can be flexibly extended to the conditions where human-labeled references are
available. Last but not least, we collect VATEX-EVAL and ActivityNet-FOIl
datasets to systematically evaluate the existing metrics. VATEX-EVAL
experiments demonstrate that EMScore has higher human correlation and lower
reference dependency. ActivityNet-FOIL experiment verifies that EMScore can
effectively identify "hallucinating" captions. The datasets will be released to
facilitate the development of video captioning metrics. The code is available
at: this https URL

本研究提出一种基于嵌入匹配的视频字幕参照无关评估度量 EMS，结合粗细粒度的视频字幕匹配、基于大规模预训练模型的视觉语言嵌入等技术，更准确地衡量视频与候选字幕之间的相似度，并在 VATEX-EVAL 数据上进行了实验验证。