Commonsense reasoning is one of the key problems in natural language
processing, but the relative scarcity of labeled data holds back the progress
for languages other than English. Pretrained cross-lingual models are a source
of powerful language-agnostic representations, yet their inherent reasoning
capabilities are still actively studied. In this work, we design a simple
approach to commonsense reasoning which trains a linear classifier with weights
of multi-head attention as features. To evaluate this approach, we create a
multilingual Winograd Schema corpus by processing several datasets from prior
work within a standardized pipeline and measure cross-lingual generalization
ability in terms of out-of-sample performance. The method performs
competitively with recent supervised and unsupervised approaches for
commonsense reasoning, even when applied to other languages in a zero-shot
manner. Also, we demonstrate that most of the performance is given by the same
small subset of attention heads for all studied languages, which provides
evidence of universal reasoning capabilities in multilingual encoders.

本研究提出一个新的简单方法，利用预先训练的跨语言模型来进行常识推理，并在多语言 Winograd 模式语料库上进行了评估，结果表明即使在其他语言中零样本情况下，该方法具有与监督和非监督方法相当的性能表现，并提供了多语言编码器普遍推理能力的证据。