Transformer-based Neural Language Models achieve state-of-the-art performance
on various natural language processing tasks. However, an open question is the
extent to which these models rely on word-order/syntactic or word
co-occurrence/topic-based information when processing natural language. This
work contributes to this debate by addressing the question of whether these
models primarily use topic as a signal, by exploring the relationship between
Transformer-based models' (BERT and RoBERTa's) performance on a range of
probing tasks in English, from simple lexical tasks such as sentence length
prediction to complex semantic tasks such as idiom token identification, and
the sensitivity of these tasks to the topic information. To this end, we
propose a novel probing method which we call topic-aware probing. Our initial
results indicate that Transformer-based models encode both topic and non-topic
information in their intermediate layers, but also that the facility of these
models to distinguish idiomatic usage is primarily based on their ability to
identify and encode topic. Furthermore, our analysis of these models'
performance on other standard probing tasks suggests that tasks that are
relatively insensitive to the topic information are also tasks that are
relatively difficult for these models.

Transformer-based 神经语言模型在自然语言处理任务中取得了最先进的性能，但一个开放问题是这些模型在处理自然语言时在词序 / 句法、词共现 / 主题相关信息中所依赖的程度。本文通过研究基于 Transformer 模型（BERT 和 RoBERTa）在英文一系列探测任务中的表现，从简单词汇任务如句子长度预测到复杂语义任务如成语标记识别，探讨了这个问题，以此加入对这一争论的贡献，并提出了一种新的探测方法 —— 主题感知探测。初步结果显示，Transformer 模型在其中间层既编码了主题信息又编码了非主题信息，但理解习惯用法的能力主要基于它们的识别和编码主题能力。此外，对这些模型在其他标准探测任务上的表现的分析表明，对主题信息相对不敏感的任务也是对这些模型相对困难的任务。