While the situation has improved for text-only models, it again seems to be
the case currently that multimodal (text and image) models develop faster than
ways to evaluate them. In this paper, we bring a recently developed evaluation
paradigm from text models to multimodal models, namely evaluation through the
goal-oriented game (self) play, complementing reference-based and
preference-based evaluation. Specifically, we define games that challenge a
model's capability to represent a situation from visual information and align
such representations through dialogue. We find that the largest closed models
perform rather well on the games that we define, while even the best
open-weight models struggle with them. On further analysis, we find that the
exceptional deep captioning capabilities of the largest models drive some of
the performance. There is still room to grow for both kinds of models, ensuring
the continued relevance of the benchmark.

我们将一种最近发展起来的评估模型应用于多模态模型，通过目标导向的游戏来评估模型的性能，我们发现最大的闭合模型在我们定义的游戏中表现相当好，而最好的开放式模型却困难重重。进一步分析发现，最大模型的卓越深层描述能力推动了一些性能。两种模型均有进一步发展的空间，保证了基准的持续重要性。