Humans perceive and comprehend different visual properties of an object based
on specific contexts. For instance, we know that a banana turns brown ``when it
becomes rotten,'' whereas it appears green ``when it is unripe.'' Previous
studies on probing visual commonsense knowledge have primarily focused on
examining language models' understanding of typical properties (e.g., colors
and shapes) of objects. We present WinoViz, a text-only evaluation dataset,
consisting of 1,380 examples that probe the reasoning abilities of language
models regarding variant visual properties of objects under different contexts
or states. Our task is challenging since it requires pragmatic reasoning
(finding intended meanings) and visual knowledge reasoning. We also present
multi-hop data, a more challenging version of our data, which requires
multi-step reasoning chains to solve our task. In our experimental analysis,
our findings are: a) Large language models such as GPT-4 demonstrate effective
performance, but when it comes to multi-hop data, their performance is
significantly degraded. b) Large models perform well on pragmatic reasoning,
but visual knowledge reasoning is a bottleneck in our task. c) Vision-language
models outperform their language-model counterparts. d) A model with
machine-generated images performs poorly in our task. This is due to the poor
quality of the generated images.

人们在特定语境下会感知和理解物体的不同视觉属性。以香蕉为例，它腐烂时变成棕色，未成熟时是绿色。我们呈现了 WinoViz，一个仅包含文本的评估数据集，包括 1,380 个示例，以测试语言模型在不同语境或状态下理解物体的变体视觉属性的推理能力。我们的任务具有挑战性，因为它需要实用推理和视觉知识推理。我们还呈现了多跳数据，这是我们数据的一种更具挑战性的版本，需要多步推理链来解决我们的任务。在我们的实验分析中，我们的发现是：a）如 GPT-4 等大型语言模型表现出良好的性能，但当涉及到多跳数据时，它们的性能显著下降。b）大型模型在实用推理方面表现较好，但在我们的任务中，视觉知识推理是一个瓶颈。c）视觉语言模型胜过它们的仅语言模型对应物。d）具有机器生成图像的模型在我们的任务中表现不佳，这是由于生成图像的质量较差。