Contextualised word vectors obtained via pre-trained language models encode a
variety of knowledge that has already been exploited in applications.
Complementary to these language models are probabilistic topic models that
learn thematic patterns from the text. Recent work has demonstrated that
conducting clustering on the word-level contextual representations from a
language model emulates word clusters that are discovered in latent topics of
words from Latent Dirichlet Allocation. The important question is how such
topical word clusters are automatically formed, through clustering, in the
language model when it has not been explicitly designed to model latent topics.
To address this question, we design different probe experiments. Using BERT and
DistilBERT, we find that the attention framework plays a key role in modelling
such word topic clusters. We strongly believe that our work paves way for
further research into the relationships between probabilistic topic models and
pre-trained language models.

通过设计不同的探针实验，使用 BERT 和 DistilBERT，我们发现注意力框架在模拟主题词簇时发挥了关键作用，这为探讨概率主题模型和预训练语言模型之间的关系铺平了道路。