Interleaved text-and-image generation has been an intriguing research
direction, where the models are required to generate both images and text
pieces in an arbitrary order. Despite the emerging advancements in interleaved
generation, the progress in its evaluation still significantly lags behind.
Existing evaluation benchmarks do not support arbitrarily interleaved images
and text for both inputs and outputs, and they only cover a limited number of
domains and use cases. Also, current works predominantly use similarity-based
metrics which fall short in assessing the quality in open-ended scenarios. To
this end, we introduce InterleavedBench, the first benchmark carefully curated
for the evaluation of interleaved text-and-image generation. InterleavedBench
features a rich array of tasks to cover diverse real-world use cases. In
addition, we present InterleavedEval, a strong reference-free metric powered by
GPT-4o to deliver accurate and explainable evaluation. We carefully define five
essential evaluation aspects for InterleavedEval, including text quality,
perceptual quality, image coherence, text-image coherence, and helpfulness, to
ensure a comprehensive and fine-grained assessment. Through extensive
experiments and rigorous human evaluation, we show that our benchmark and
metric can effectively evaluate the existing models with a strong correlation
with human judgments surpassing previous reference-based metrics. We also
provide substantial findings and insights to foster future research in
interleaved generation and its evaluation.

我们介绍了 InterleavedBench 作为第一个精心策划的用于评估混合文本图像生成的基准，并引入了使用 GPT-4o 提供准确且可解释的评估的强无参考度量 InterleavedEval。通过广泛的实验和严格的人工评估，我们表明我们的基准和度量可以有效评估现有模型，并与以前的基于参考的度量具有较强的相关性。