Since Pretrained Language Models (PLMs) are the cornerstone of the most recent Information Retrieval (IR) models, the way they encode semantic knowledge is particularly important. However, little attention has been given to studying the PLMs' capability to capture hierarchical semantic knowledge. Traditionally, evaluating such knowledge encoded in PLMs relies on their performance on a task-dependent evaluation approach based on proxy tasks, such as hypernymy detection. Unfortunately, this approach potentially ignores other implicit and complex taxonomic relations. In this work, we propose a task-agnostic evaluation method able to evaluate to what extent PLMs can capture complex taxonomy relations, such as ancestors and siblings. The evaluation is based on intrinsic properties that capture the hierarchical nature of taxonomies. Our experimental evaluation shows that the lexico-semantic knowledge implicitly encoded in PLMs does not always capture hierarchical relations. We further demonstrate that the proposed properties can be injected into PLMs to improve their understanding of hierarchy. Through evaluations on taxonomy reconstruction, hypernym discovery and reading comprehension tasks, we show that the knowledge about hierarchy is moderately but not systematically transferable across tasks.

这篇论文提出了一种能够评估预训练语言模型在捕捉复杂的分类关系方面能力的无任务评估方法，并通过实验证明预训练语言模型隐式编码的词汇-语义知识并未始终捕捉到层次关系。此外，作者进一步展示了注入这种评估方法能够改善预训练语言模型对层次关系的理解，并通过对分类体系重构、上位词发现和阅读理解任务的评估表明，层次结构的知识在不同任务间有一定的但不系统的传递性。

探索具有层次性质的预训练语言模型