In this paper, we present the LingOly benchmark, a novel benchmark for
advanced reasoning abilities in large language models. Using challenging
Linguistic Olympiad puzzles, we evaluate (i) capabilities for in-context
identification and generalisation of linguistic patterns in very low-resource
or extinct languages, and (ii) abilities to follow complex task instructions.
The LingOly benchmark covers more than 90 mostly low-resource languages,
minimising issues of data contamination, and contains 1,133 problems across 6
formats and 5 levels of human difficulty. We assess performance with both
direct accuracy and comparison to a no-context baseline to penalise
memorisation. Scores from 11 state-of-the-art LLMs demonstrate the benchmark to
be challenging, and models perform poorly on the higher difficulty problems. On
harder problems, even the top model only achieved 35.3% accuracy, 21.7%
improvement over the no-context baseline. Large closed models typically
outperform open models, and in general, the higher resource the language, the
better the scores. These results indicate, in absence of memorisation, true
multi-step out-of-domain reasoning remains a challenge for current language
models.

通过 LingOly 基准测试，我们评估了大型语言模型在上下文识别、语言模式的泛化能力以及执行复杂任务指令的能力，并发现在没有记忆的情况下，当前语言模型在真正的多步骤跨领域推理方面仍存在挑战。