We introduce modeLing, a novel benchmark of Linguistics Olympiad-style
puzzles which tests few-shot reasoning in AI systems. Solving these puzzles
necessitates inferring aspects of a language's grammatical structure from a
small number of examples. Such puzzles provide a natural testbed for language
models, as they require compositional generalization and few-shot inductive
reasoning. Consisting solely of new puzzles written specifically for this work,
modeLing has no risk of appearing in the training data of existing AI systems:
this ameliorates the risk of data leakage, a potential confounder for many
prior evaluations of reasoning. Evaluating several large open source language
models and GPT on our benchmark, we observe non-negligible accuracy,
demonstrating few-shot emergent reasoning ability which cannot merely be
attributed to shallow memorization. However, imperfect model performance
suggests that modeLing can be used to measure further progress in linguistic
reasoning.

我们介绍了 modeLing，这是一种测试人工智能系统中少样本推理能力的新型 Linguistics Olympiad-style 谜题基准。通过从少数例子中推断一种语言的语法结构的各个方面，解决这些谜题需要一定的推理能力。modeLing 是专门为本研究编写的全新谜题，不会出现在现有 AI 系统的训练数据中，减少了数据泄漏对推理评估的潜在混淆因素。通过在我们的基准测试中评估多个大型开源语言模型和 GPT，我们观察到相当高的准确性，表明具备少样本新兴推理能力，不能仅归因于浅层记忆。然而，不完美的模型表现表明 modeLing 可以用来衡量语言推理的进一步进展。