Large language models are known to suffer from the hallucination problem in
that they are prone to output statements that are false or inconsistent,
indicating a lack of knowledge. A proposed solution to this is to provide the
model with additional data modalities that complements the knowledge obtained
through text. We investigate the use of visual data to complement the knowledge
of large language models by proposing a method for evaluating visual knowledge
transfer to text for uni- or multimodal language models. The method is based on
two steps, 1) a novel task querying for knowledge of memory colors, i.e.
typical colors of well-known objects, and 2) filtering of model training data
to clearly separate knowledge contributions. Additionally, we introduce a model
architecture that involves a visual imagination step and evaluate it with our
proposed method. We find that our method can successfully be used to measure
visual knowledge transfer capabilities in models and that our novel model
architecture shows promising results for leveraging multimodal knowledge in a
unimodal setting.

探究使用视觉数据来补充大型语言模型知识的方法，并提出一种基于 memory colors 任务查询和模型训练数据过滤的方法，以用于测量模型的视觉知识传输能力，并介绍了一种涉及视觉想象步骤的模型架构，并发现我们的方法可以成功用于衡量模型的视觉知识传输能力，而我们的新型模型架构在单模态环境中利用多模态知识具有良好的结果。