The advent of powerful neural classifiers has increased interest in problems
that require both learning and reasoning. These problems are critical for
understanding important properties of models, such as trustworthiness,
generalization, interpretability, and compliance to safety and structural
constraints. However, recent research observed that tasks requiring both
learning and reasoning on background knowledge often suffer from reasoning
shortcuts (RSs): predictors can solve the downstream reasoning task without
associating the correct concepts to the high-dimensional data. To address this
issue, we introduce rsbench, a comprehensive benchmark suite designed to
systematically evaluate the impact of RSs on models by providing easy access to
highly customizable tasks affected by RSs. Furthermore, rsbench implements
common metrics for evaluating concept quality and introduces novel formal
verification procedures for assessing the presence of RSs in learning tasks.
Using rsbench, we highlight that obtaining high quality concepts in both purely
neural and neuro-symbolic models is a far-from-solved problem. rsbench is
available at: this https URL

强大的神经分类器的出现增加了对既需要学习又需要推理的问题的兴趣，该研究论文介绍了一个包含任务评估、概念质量评估和推理快捷方式验证的综合基准套件，旨在系统地评估推理快捷方式对模型的影响。

系统评估推理捷径的基准套件

A Benchmark Suite for Systematically Evaluating Reasoning Shortcuts

The field of neuro-symbolic artificial intelligence (NeSy), which combines
learning and reasoning, has recently experienced significant growth. There now
are a wide variety of NeSy frameworks, each with its own specific language for
expressing background knowledge and how to relate it to neural networks. This
heterogeneity hinders accessibility for newcomers and makes comparing different
NeSy frameworks challenging. We propose a unified language for NeSy, which we
call ULLER, a Unified Language for LEarning and Reasoning. ULLER encompasses a
wide variety of settings, while ensuring that knowledge described in it can be
used in existing NeSy systems. ULLER has a neuro-symbolic first-order syntax
for which we provide example semantics including classical, fuzzy, and
probabilistic logics. We believe ULLER is a first step towards making NeSy
research more accessible and comparable, paving the way for libraries that
streamline training and evaluation across a multitude of semantics, knowledge
bases, and NeSy systems.

提出了一种名为 ULLER 的统一语言，用于学习和推理的神经符号人工智能（NeSy），该语言可以让新手更容易进入该领域并比较不同的 NeSy 框架和系统。

ULLER: 学习和推理的统一语言

ULLER: A Unified Language for Learning and Reasoning

The integration of learning and reasoning is high on the research agenda in
AI. Nevertheless, there is only a little attention to use existing background
knowledge for reasoning about partially observed scenes to answer questions
about the scene. Yet, we as humans use such knowledge frequently to infer
plausible answers to visual questions (by eliminating all inconsistent ones).
Such knowledge often comes in the form of constraints about objects and it
tends to be highly domain or environment-specific. We contribute a novel
benchmark called CLEVR-POC for reasoning-intensive visual question answering
(VQA) in partially observable environments under constraints. In CLEVR-POC,
knowledge in the form of logical constraints needs to be leveraged to generate
plausible answers to questions about a hidden object in a given partial scene.
For instance, if one has the knowledge that all cups are colored either red,
green or blue and that there is only one green cup, it becomes possible to
deduce the color of an occluded cup as either red or blue, provided that all
other cups, including the green one, are observed. Through experiments, we
observe that the low performance of pre-trained vision language models like
CLIP (~ 22%) and a large language model (LLM) like GPT-4 (~ 46%) on CLEVR-POC
ascertains the necessity for frameworks that can handle reasoning-intensive
tasks where environment-specific background knowledge is available and crucial.
Furthermore, our demonstration illustrates that a neuro-symbolic model, which
integrates an LLM like GPT-4 with a visual perception network and a formal
logical reasoner, exhibits exceptional performance on CLEVR-POC.

CLEVR-POC 是一个基于部分可观察环境下的逻辑约束的推理密集型视觉问答基准，通过集成 GPT-4 大型语言模型、视觉感知网络和形式逻辑推理器，展示了在可用且至关重要的环境特定背景知识下实现卓越性能的神经符号模型。