We introduce modeLing, a novel benchmark of Linguistics Olympiad-style
puzzles which tests few-shot reasoning in AI systems. Solving these puzzles
necessitates inferring aspects of a language's grammatical structure from a
small number of examples. Such puzzles provide a natural testbed for language
models, as they require compositional generalization and few-shot inductive
reasoning. Consisting solely of new puzzles written specifically for this work,
modeLing has no risk of appearing in the training data of existing AI systems:
this ameliorates the risk of data leakage, a potential confounder for many
prior evaluations of reasoning. Evaluating several large open source language
models and GPT on our benchmark, we observe non-negligible accuracy,
demonstrating few-shot emergent reasoning ability which cannot merely be
attributed to shallow memorization. However, imperfect model performance
suggests that modeLing can be used to measure further progress in linguistic
reasoning.

我们介绍了 modeLing，这是一种测试人工智能系统中少样本推理能力的新型 Linguistics Olympiad-style 谜题基准。通过从少数例子中推断一种语言的语法结构的各个方面，解决这些谜题需要一定的推理能力。modeLing 是专门为本研究编写的全新谜题，不会出现在现有 AI 系统的训练数据中，减少了数据泄漏对推理评估的潜在混淆因素。通过在我们的基准测试中评估多个大型开源语言模型和 GPT，我们观察到相当高的准确性，表明具备少样本新兴推理能力，不能仅归因于浅层记忆。然而，不完美的模型表现表明 modeLing 可以用来衡量语言推理的进一步进展。

modeLing: 用于测试语言模型语言推理的新颖数据集

modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language  Models

Math word problem (MWP) solving requires generating a reasoning path based on
a given problem description that often contains irrelevant conditions. Existing
chain-of-thought (CoT) prompting methods elicited multi-step reasoning
abilities of large language models (LLMs) to solve MWPs. However, they were
seriously confused by the irrelevant conditions, resulting in low accuracy. In
this paper, we propose a novel approach named I$^3$C that instructs LLMs to
identify and ignore irrelevant conditions. It identifies a set of irrelevant
condition candidates that have a weak semantic relevance with the question.
Then it prompts LLMs to verify the irrelevant conditions. Lastly it instructs
the LLMs with the verification on relevant and irrelevant conditions to avoid
confusion and improve reasoning paths. Moreover, we propose to select (problem,
reasoning paths) pairs as demonstrations to enhance I$^3$C with few-shot
reasoning. We develop I$^3$C-Select that selects the most confusing problems
based on the semantic relevance measurement. We conduct extensive experiments
on eight MWP datasets. I$^3$C can be combined with any CoT prompting methods to
improve the performance of solving MWPs. Notably, with GPT-3.5-Turbo and
I$^3$C-Select, we achieve an accuracy of 96.0 and 94.1 on GSM-IC2-1K and
GSM-ICM-1K, respectively, significantly outperforming the state-of-the-art
few-shot prompting method Complex-CoT by +11.7 and +11.1. Our implementation is
made publicly available at this https URL

通过提出一种名为 I$^3$C 的新方法，指导大型语言模型识别和忽略无关条件从而提高解决数学问题的准确性和推理路径

指导大型语言模型识别和忽略无关条件

Instructing Large Language Models to Identify and Ignore Irrelevant  Conditions

We introduce Bongard-OpenWorld, a new benchmark for evaluating real-world
few-shot reasoning for machine vision. It originates from the classical Bongard
Problems (BPs): Given two sets of images (positive and negative), the model
needs to identify the set that query images belong to by inducing the visual
concepts, which is exclusively depicted by images from the positive set. Our
benchmark inherits the few-shot concept induction of the original BPs while
adding the two novel layers of challenge: 1) open-world free-form concepts, as
the visual concepts in Bongard-OpenWorld are unique compositions of terms from
an open vocabulary, ranging from object categories to abstract visual
attributes and commonsense factual knowledge; 2) real-world images, as opposed
to the synthetic diagrams used by many counterparts. In our exploration,
Bongard-OpenWorld already imposes a significant challenge to current few-shot
reasoning algorithms. We further investigate to which extent the recently
introduced Large Language Models (LLMs) and Vision-Language Models (VLMs) can
solve our task, by directly probing VLMs, and combining VLMs and LLMs in an
interactive reasoning scheme. We even designed a neuro-symbolic reasoning
approach that reconciles LLMs & VLMs with logical reasoning to emulate the
human problem-solving process for Bongard Problems. However, none of these
approaches manage to close the human-machine gap, as the best learner achieves
64% accuracy while human participants easily reach 91%. We hope
Bongard-OpenWorld can help us better understand the limitations of current
visual intelligence and facilitate future research on visual agents with
stronger few-shot visual reasoning capabilities.

我们引入了 Bongard-OpenWorld，这是一个用于评估机器视觉领域的真实世界少样本推理的新基准。它继承了经典的 Bongard 问题的少样本概念归纳，并添加了两个新的挑战层：一是开放世界的自由形式概念，二是真实世界的图像。然而，目前的学习算法仍无法达到人类的准确度，我们希望 Bongard-OpenWorld 能帮助我们更好地理解当前视觉智能的局限性，并促进未来对具有较强少样本视觉推理能力的视觉智能体的研究。