Current foundation models exhibit impressive capabilities when prompted
either with text only or with both image and text inputs. But do their
capabilities change depending on the input modality? In this work, we propose
$\textbf{IsoBench}$, a benchmark dataset containing problems from four major
areas: math, science, algorithms, and games. Each example is presented with
multiple $\textbf{isomorphic representations}$ of inputs, such as visual,
textual, and mathematical presentations. IsoBench provides fine-grained
feedback to diagnose performance gaps caused by the form of the representation.
Across various foundation models, we observe that on the same problem, models
have a consistent preference towards textual representations. Most prominently,
when evaluated on all IsoBench problems, Claude-3 Opus performs 28.7 points
worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.7
points worse and Gemini Pro is 14.9 points worse. Finally, we present two
prompting techniques, $\textit{IsoCombination}$ and $\textit{IsoScratchPad}$,
which improve model performance by considering combinations of, and
translations between, different input representations.

当前的基础模型在以文本或图像和文本输入作为提示时都展现出令人印象深刻的能力。然而，它们的能力是否因输入模态的不同而改变呢？本研究提出了一种名为 IsoBench 的基准数据集，其中包含了来自数学、科学、算法和游戏四个主要领域的问题。每个示例都以多种同构表示形式呈现，如视觉、文本和数学表达。IsoBench 提供了细粒度的反馈，以诊断由表示形式引起的性能差距。在各种基础模型中，我们发现在同一个问题上，模型对文本表示有一致的偏好。尤其是，在对所有 IsoBench 问题进行评估时，Claude-3 Opus 模型在提供图像而不是文本时表现较差，差距为 28.7 分；同样，GPT-4 Turbo 差距为 18.7 分，Gemini Pro 差距为 14.9 分。最后，我们提出了两种提示技术，即 IsoCombination 和 IsoScratchPad，通过考虑不同输入表示之间的组合和转换，提高了模型的性能。