Cryptic crosswords are puzzles that rely not only on general knowledge but
also on the solver's ability to manipulate language on different levels and
deal with various types of wordplay. Previous research suggests that solving
such puzzles is a challenge even for modern NLP models. However, the abilities
of large language models (LLMs) have not yet been tested on this task. In this
paper, we establish the benchmark results for three popular LLMs -- LLaMA2,
Mistral, and ChatGPT -- showing that their performance on this task is still
far from that of humans.

利用三种大型语言模型对神秘填字游戏进行基准测试，结果显示它们在这个任务上的表现远远低于人类。

LLM 是否是出色的密码填字游戏解答者？

Are LLMs Good Cryptic Crossword Solvers?

Current NLP datasets targeting ambiguity can be solved by a native speaker
with relative ease. We present Cryptonite, a large-scale dataset based on
cryptic crosswords, which is both linguistically complex and naturally sourced.
Each example in Cryptonite is a cryptic clue, a short phrase or sentence with a
misleading surface reading, whose solving requires disambiguating semantic,
syntactic, and phonetic wordplays, as well as world knowledge. Cryptic clues
pose a challenge even for experienced solvers, though top-tier experts can
solve them with almost 100% accuracy. Cryptonite is a challenging task for
current models; fine-tuning T5-Large on 470k cryptic clues achieves only 7.6%
accuracy, on par with the accuracy of a rule-based clue solver (8.6%).

提出一个基于密码填字游戏的大规模数据集 Cryptonite，要求解出由语义、语法、音位上的文字玩偶和世界知识构成的密语线索，该任务难度高，当前模型的表现只有 7.6%，与基于规则的求解器的表现相当。