Many applications need access to background knowledge about how different
concepts and entities are related. Although Knowledge Graphs (KG) and Large
Language Models (LLM) can address this need to some extent, KGs are inevitably
incomplete and their relational schema is often too coarse-grained, while LLMs
are inefficient and difficult to control. As an alternative, we propose to
extract relation embeddings from relatively small language models. In
particular, we show that masked language models such as RoBERTa can be
straightforwardly fine-tuned for this purpose, using only a small amount of
training data. The resulting model, which we call RelBERT, captures relational
similarity in a surprisingly fine-grained way, allowing us to set a new
state-of-the-art in analogy benchmarks. Crucially, RelBERT is capable of
modelling relations that go well beyond what the model has seen during
training. For instance, we obtained strong results on relations between named
entities with a model that was only trained on lexical relations between
concepts, and we observed that RelBERT can recognise morphological analogies
despite not being trained on such examples. Overall, we find that RelBERT
significantly outperforms strategies based on prompting language models that
are several orders of magnitude larger, including recent GPT-based models and
open source models.

我们提出了一种使用相对较小的语言模型从文本中提取关系嵌入的方法，这种方法可以在关系相似性方面取得出色的结果，并且在关键词和模型性能方面显著优于其他基于提示的语言模型。