Recently, open-domain text-to-video (T2V) generation models have made
remarkable progress. However, the promising results are mainly shown by the
qualitative cases of generated videos, while the quantitative evaluation of T2V
models still faces two critical problems. Firstly, existing studies lack
fine-grained evaluation of T2V models on different categories of text prompts.
Although some benchmarks have categorized the prompts, their categorization
either only focuses on a single aspect or fails to consider the temporal
information in video generation. Secondly, it is unclear whether the automatic
evaluation metrics are consistent with human standards. To address these
problems, we propose FETV, a benchmark for Fine-grained Evaluation of
Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based
on three orthogonal aspects: the major content, the attributes to control and
the prompt complexity. FETV is also temporal-aware, which introduces several
temporal categories tailored for video generation. Based on FETV, we conduct
comprehensive manual evaluations of four representative T2V models, revealing
their pros and cons on different categories of prompts from different aspects.
We also extend FETV as a testbed to evaluate the reliability of automatic T2V
metrics. The multi-aspect categorization of FETV enables fine-grained analysis
of the metrics' reliability in different scenarios. We find that existing
automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human
evaluation. To address this problem, we explore several solutions to improve
CLIPScore and FVD, and develop two automatic metrics that exhibit significant
higher correlation with humans than existing metrics. Benchmark page:
this https URL

我们提出了 FETV，一个用于细粒度评估文本到视频生成的基准，它基于三个正交方面对提示进行分类，并且是与时间相关的，通过该基准我们对四个代表性的 T2V 模型进行了全面的手动评估，发现现有的自动评估指标与人工评估的相关性较差，并提出了两个新的自动评估指标与人工评估相关性更高。