The integration of learning and reasoning is high on the research agenda in
AI. Nevertheless, there is only a little attention to use existing background
knowledge for reasoning about partially observed scenes to answer questions
about the scene. Yet, we as humans use such knowledge frequently to infer
plausible answers to visual questions (by eliminating all inconsistent ones).
Such knowledge often comes in the form of constraints about objects and it
tends to be highly domain or environment-specific. We contribute a novel
benchmark called CLEVR-POC for reasoning-intensive visual question answering
(VQA) in partially observable environments under constraints. In CLEVR-POC,
knowledge in the form of logical constraints needs to be leveraged to generate
plausible answers to questions about a hidden object in a given partial scene.
For instance, if one has the knowledge that all cups are colored either red,
green or blue and that there is only one green cup, it becomes possible to
deduce the color of an occluded cup as either red or blue, provided that all
other cups, including the green one, are observed. Through experiments, we
observe that the low performance of pre-trained vision language models like
CLIP (~ 22%) and a large language model (LLM) like GPT-4 (~ 46%) on CLEVR-POC
ascertains the necessity for frameworks that can handle reasoning-intensive
tasks where environment-specific background knowledge is available and crucial.
Furthermore, our demonstration illustrates that a neuro-symbolic model, which
integrates an LLM like GPT-4 with a visual perception network and a formal
logical reasoner, exhibits exceptional performance on CLEVR-POC.

CLEVR-POC 是一个基于部分可观察环境下的逻辑约束的推理密集型视觉问答基准，通过集成 GPT-4 大型语言模型、视觉感知网络和形式逻辑推理器，展示了在可用且至关重要的环境特定背景知识下实现卓越性能的神经符号模型。