Rationales in the form of manually annotated input spans usually serve as
ground truth when evaluating explainability methods in NLP. They are, however,
time-consuming and often biased by the annotation process. In this paper, we
debate whether human gaze, in the form of webcam-based eye-tracking recordings,
poses a valid alternative when evaluating importance scores. We evaluate the
additional information provided by gaze data, such as total reading times, gaze
entropy, and decoding accuracy with respect to human rationale annotations. We
compare WebQAmGaze, a multilingual dataset for information-seeking QA, with
attention and explainability-based importance scores for 4 different
multilingual Transformer-based language models (mBERT, distil-mBERT, XLMR, and
XLMR-L) and 3 languages (English, Spanish, and German). Our pipeline can easily
be applied to other tasks and languages. Our findings suggest that gaze data
offers valuable linguistic insights that could be leveraged to infer task
difficulty and further show a comparable ranking of explainability methods to
that of human rationales.

人的凝视数据在评估重要性评分时作为一种有效的替代品，能够提供有价值的语言洞察力，并且在解释方法的排名上与人工评估的理据相媲美。

基于网络摄像头的凝视数据评估作为人类理由注释的替代方法

Evaluating Webcam-based Gaze Data as an Alternative for Human Rationale  Annotations

Multilingual Transformer-based language models, usually pretrained on more
than 100 languages, have been shown to achieve outstanding results in a wide
range of cross-lingual transfer tasks. However, it remains unknown whether the
optimization for different languages conditions the capacity of the models to
generalize over syntactic structures, and how languages with syntactic
phenomena of different complexity are affected. In this work, we explore the
syntactic generalization capabilities of the monolingual and multilingual
versions of BERT and RoBERTa. More specifically, we evaluate the syntactic
generalization potential of the models on English and Spanish tests, comparing
the syntactic abilities of monolingual and multilingual models on the same
language (English), and of multilingual models on two different languages
(English and Spanish). For English, we use the available SyntaxGym test suite;
for Spanish, we introduce SyntaxGymES, a novel ensemble of targeted syntactic
tests in Spanish, designed to evaluate the syntactic generalization
capabilities of language models through the SyntaxGym online platform.

本研究探讨了 Bert 和 RoBERTa 等单语和多语版本的句法泛化能力，使用 SyntaxGym 测试套件和其在西班牙语中的替代品 SyntaxGymES 对英语和西班牙语进行了测试。