Recent text-to-video (T2V) technology advancements, as demonstrated by models
such as Gen2, Pika, and Sora, have significantly broadened its applicability
and popularity. Despite these strides, evaluating these models poses
substantial challenges. Primarily, due to the limitations inherent in automatic
metrics, manual evaluation is often considered a superior method for assessing
T2V generation. However, existing manual evaluation protocols face
reproducibility, reliability, and practicality issues. To address these
challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE)
protocol, a comprehensive and standardized protocol for T2V models. The T2VHE
protocol includes well-defined metrics, thorough annotator training, and an
effective dynamic evaluation module. Experimental results demonstrate that this
protocol not only ensures high-quality annotations but can also reduce
evaluation costs by nearly 50%. We will open-source the entire setup of the
T2VHE protocol, including the complete protocol workflow, the dynamic
evaluation component details, and the annotation interface code. This will help
communities establish more sophisticated human assessment protocols.

该研究介绍了一种用于评估文本到视频模型的全面标准化协议 ——T2VHE 协议，包括明确定义的度量标准、细致的标注员培训和有效的动态评估模块，实验证明该协议不仅确保了高质量的注释，还可以将评估成本降低近 50%。