Searching troves of videos with textual descriptions is a core multimodal
retrieval task. Owing to the lack of a purpose-built dataset for text-to-video
retrieval, video captioning datasets have been re-purposed to evaluate models
by (1) treating captions as positive matches to their respective videos and (2)
assuming all other videos to be negatives. However, this methodology leads to a
fundamental flaw during evaluation: since captions are marked as relevant only
to their original video, many alternate videos also match the caption, which
introduces false-negative caption-video pairs. We show that when these false
negatives are corrected, a recent state-of-the-art model gains 25\% recall
points -- a difference that threatens the validity of the benchmark itself. To
diagnose and mitigate this issue, we annotate and release 683K additional
caption-video pairs. Using these, we recompute effectiveness scores for three
models on two standard benchmarks (MSR-VTT and MSVD). We find that (1) the
recomputed metrics are up to 25\% recall points higher for the best models, (2)
these benchmarks are nearing saturation for Recall@10, (3) caption length
(generality) is related to the number of positives, and (4) annotation costs
can be mitigated through sampling. We recommend retiring these benchmarks in
their current form, and we make recommendations for future text-to-video
retrieval benchmarks.

通过修正误标的负样本，我们评测了三种模型在两个标准测试集上的表现，发现在最佳模型上，修正后的指标提升了 25% 以上。此外我们发现测试集的 recall@10 已经接近饱和，同时我们推荐以样本抽样方式缓解标注数据成本，对未来的 text-to-video retrieval 基准测试提出了建议。