Generative AI has made remarkable strides to revolutionize fields such as
image and video generation. These advancements are driven by innovative
algorithms, architecture, and data. However, the rapid proliferation of
generative models has highlighted a critical gap: the absence of trustworthy
evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc
often fail to capture the nuanced quality and user satisfaction associated with
generative outputs. This paper proposes an open platform GenAI-Arena to
evaluate different image and video generative models, where users can actively
participate in evaluating these models. By leveraging collective user feedback
and votes, GenAI-Arena aims to provide a more democratic and accurate measure
of model performance. It covers three arenas for text-to-image generation,
text-to-video generation, and image editing respectively. Currently, we cover a
total of 27 open-source generative models. GenAI-Arena has been operating for
four months, amassing over 6000 votes from the community. We describe our
platform, analyze the data, and explain the statistical methods for ranking the
models. To further promote the research in building model-based evaluation
metrics, we release a cleaned version of our preference data for the three
tasks, namely GenAI-Bench. We prompt the existing multi-modal models like
Gemini, GPT-4o to mimic human voting. We compute the correlation between model
voting with human voting to understand their judging abilities. Our results
show existing multimodal models are still lagging in assessing the generated
visual content, even the best model GPT-4o only achieves a Pearson correlation
of 0.22 in the quality subscore, and behaves like random guessing in others.

通过开源平台 GenAI-Arena 和统计方法，该研究论文提出一种评估图像和视频生成模型的方法，以更准确地衡量模型性能，并发现现有的多模态模型在评估生成的视觉内容方面存在不足。