We investigate the extent to which verb alternation classes, as described by
Levin (1993), are encoded in the embeddings of Large Pre-trained Language
Models (PLMs) such as BERT, RoBERTa, ELECTRA, and DeBERTa using selectively
constructed diagnostic classifiers for word and sentence-level prediction
tasks. We follow and expand upon the experiments of Kann et al. (2019), which
aim to probe whether static embeddings encode frame-selectional properties of
verbs. At both the word and sentence level, we find that contextual embeddings
from PLMs not only outperform non-contextual embeddings, but achieve
astonishingly high accuracies on tasks across most alternation classes.
Additionally, we find evidence that the middle-to-upper layers of PLMs achieve
better performance on average than the lower layers across all probing tasks.

本研究通过选择性构建的诊断分类器，对 Bert、Roberta、Electra 和 DeBERTa 等大型预训练语言模型的嵌入进行了研究。研究发现，PLM 的上下文嵌入不仅在单词和句子级别上优于非上下文嵌入，而且在大多数替换类别的任务中实现了惊人的高准确性，同时，我们发现，PLM 的中至上层在所有探测任务中的表现均比下层要好。