Recent work has raised concerns about the inherent limitations of text-only
pretraining. In this paper, we first demonstrate that reporting bias, the
tendency of people to not state the obvious, is one of the causes of this
limitation, and then investigate to what extent multimodal training can
mitigate this issue. To accomplish this, we 1) generate the Color Dataset
(CoDa), a dataset of human-perceived color distributions for 521 common
objects; 2) use CoDa to analyze and compare the color distribution found in
text, the distribution captured by language models, and a human's perception of
color; and 3) investigate the performance differences between text-only and
multimodal models on CoDa. Our results show that the distribution of colors
that a language model recovers correlates more strongly with the inaccurate
distribution found in text than with the ground-truth, supporting the claim
that reporting bias negatively impacts and inherently limits text-only
training. We then demonstrate that multimodal models can leverage their visual
training to mitigate these effects, providing a promising avenue for future
research.

本文研究了文本预训练的固有局限性，提出了报告偏差作为该限制的原因之一，并分析了多模态训练在解决该问题方面的效果。研究使用了 Color Dataset 并比较了文本中的颜色分布，语言模型捕获的分布以及人类感知中的颜色分布，结果表明文本中的 reporting bias 负面影响并固有限制了单一模态训练，而多模态模型可以减轻此类影响。