The stunning qualitative improvement of recent text-to-image models has led
to their widespread attention and adoption. However, we lack a comprehensive
quantitative understanding of their capabilities and risks. To fill this gap,
we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models
(HEIM). Whereas previous evaluations focus mostly on text-image alignment and
image quality, we identify 12 aspects, including text-image alignment, image
quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,
fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios
encompassing these aspects and evaluate 26 state-of-the-art text-to-image
models on this benchmark. Our results reveal that no single model excels in all
aspects, with different models demonstrating different strengths. We release
the generated images and human evaluation results for full transparency at
this https URL and the code at
this https URL, which is integrated with the HELM
codebase.

对于最近的文本到图像模型，我们缺乏对其能力和风险的全面定量理解。为了填补这个空白，我们引入了一个新的基准，即文本到图像模型的整体评估（HEIM）。我们鉴定了 12 个方面，包括文本与图像的对齐、图像质量、美感、原创性、推理能力、知识、偏见、毒性、公平性、鲁棒性、多语性和效率。我们在这个基准上评估了 26 个最先进的文本到图像模型，结果表明没有单一模型在所有方面都表现出色，不同模型展示了不同的优势。我们透明地发布了生成的图像和人工评估结果。