Deep neural models have repeatedly proved excellent at memorizing surface
patterns from large datasets for various ML and NLP benchmarks. They struggle
to achieve human-like thinking, however, because they lack the skill of
iterative reasoning upon knowledge. To expose this problem in a new light, we
introduce a challenge on learning from small data, PuzzLing Machines, which
consists of Rosetta Stone puzzles from Linguistic Olympiads for high school
students. These puzzles are carefully designed to contain only the minimal
amount of parallel text necessary to deduce the form of unseen expressions.
Solving them does not require external information (e.g., knowledge bases,
visual signals) or linguistic expertise, but meta-linguistic awareness and
deductive skills. Our challenge contains around 100 puzzles covering a wide
range of linguistic phenomena from 81 languages. We show that both simple
statistical algorithms and state-of-the-art deep neural models perform
inadequately on this challenge, as expected. We hope that this benchmark,
available at this https URL, inspires further
efforts towards a new paradigm in NLP---one that is grounded in human-like
reasoning and understanding.

介绍了一个名为 PuzzLing Machines 的小数据学习挑战，该挑战包含来自语言奥林匹克的石碑难题，展示了目前包括最先进的深度神经网络模型在内的简单的统计算法无法完成这一挑战，这启发了新的 NLP 发展方向 -- 人类般思考的理解。