We propose a novel and challenging benchmark, AutoEval-Video, to
comprehensively evaluate large vision-language models in open-ended video
question answering. The comprehensiveness of AutoEval-Video is demonstrated in
two aspects: 1) AutoEval-Video constructs open-ended video-questions across 9
skill dimensions, addressing capabilities of perception, comprehension, and
generation. 2) AutoEval-Video contains newly collected videos that cover over
40 distinct themes. To efficiently evaluate responses to the open-ended
questions, we employ an LLM-based evaluation approach, but instead of merely
providing a reference answer, we annotate unique evaluation rules for every
single instance (video-question pair). To maximize the robustness of these
rules, we develop a novel adversarial annotation mechanism. By using
instance-specific rules as prompt, GPT-4, as an automatic evaluator, can
achieve a stable evaluation accuracy of around 97.0\%, comparable to the 94.9\%
- 97.5\% accuracy of a human evaluator. Furthermore, we assess the performance
of eight large vision-language models on AutoEval-Video. Among them,
GPT-4V(ision) significantly outperforms other models, achieving an accuracy of
32.2\%. However, there is still substantial room for improvement compared to
human accuracy of 72.8\%. By conducting an extensive case study, we uncover
several drawbacks of GPT-4V, such as limited temporal and dynamic
comprehension, and overly general responses. Code is available at
\href{https://github.com/Xiuyuan-Chen/AutoEval-Video}{\color{magenta}this https URL}.

我们提出了一个新颖且具有挑战性的基准，AutoEval-Video，以全面评估开放式视频问答中的大规模视觉语言模型。