Interpreting hierarchical structures latent in language is a key limitation
of current language models (LMs). While previous research has implicitly
leveraged these hierarchies to enhance LMs, approaches for their explicit
encoding are yet to be explored. To address this, we introduce a novel approach
to re-train transformer encoder-based LMs as Hierarchy Transformer encoders
(HiTs), harnessing the expansive nature of hyperbolic space. Our method
situates the output embedding space of pre-trained LMs within a Poincar\'e ball
with a curvature that adapts to the embedding dimension, followed by
re-training on hyperbolic cluster and centripetal losses. These losses are
designed to effectively cluster related entities (input as texts) and organise
them hierarchically. We evaluate HiTs against pre-trained and fine-tuned LMs,
focusing on their capabilities in simulating transitive inference, predicting
subsumptions, and transferring knowledge across hierarchies. The results
demonstrate that HiTs consistently outperform both pre-trained and fine-tuned
LMs in these tasks, underscoring the effectiveness and transferability of our
re-trained hierarchy encoders.

利用超几何空间重新训练语言模型中的分层转换器编码器（Hierarchy Transformer encoders，HiTs），为回归语言模型中隐含的分层结构提供了一种新的方法，通过聚类与层级组织相关实体来提高在推论、预测和跨层次知识传递等任务中的性能和可转移性。