The model's ability to understand synonymous expression is crucial in many
kinds of downstream tasks. It will make the model to better understand the
similarity between context, and more robust to the synonym substitution attack.
However, many Pretrained Language Model (PLM) lack synonym knowledge due to
limitation of small-scale synsets and PLM's pretraining objectives. In this
paper, we propose a framework called Sem4SAP to mine synsets from Open
Knowledge Graph (Open-KG) and using the mined synsets to do synonym-aware
pretraining for language models. We propose to coarsly filter the content in
Open-KG and use the frequency information to better help the clustering process
under low-resource unsupervised conditions. We expand the mined synsets by
migrating core semantics between synonymous expressions.We also propose two
novel and effective synonym-aware pre-training methods for injecting synonym
knowledge into PLMs.Extensive experiments demonstrate that Sem4SAP can
dramatically outperform the original PLMs and other baselines on ten different
tasks.

本文提出了一种叫 Sem4SAP 的框架，从 Open Knowledge Graph 中挖掘同义词集，使用这些同义词集来进行语言模型的同义词感知预训练。我们提出了两种新颖的且有效的同义词感知预训练方法，并通过大量实验证明，Sem4SAP 在十项不同任务上显著优于原始 PLMs 和其它基线模型。

Sem4SAP：基于开放知识图谱的同义词表达挖掘技术，用于语言模型的同义词感知预训练

Sem4SAP: Synonymous Expression Mining From Open Knowledge Graph For Language Model Synonym-Aware Pretraining

Open knowledge graph (KG) consists of (subject, relation, object) triples
extracted from millions of raw text. The subject and object noun phrases and
the relation in open KG have severe redundancy and ambiguity and need to be
canonicalized. Existing datasets for open KG canonicalization only provide gold
entity-level canonicalization for noun phrases. In this paper, we present
COMBO, a Complete Benchmark for Open KG canonicalization. Compared with
existing datasets, we additionally provide gold canonicalization for relation
phrases, gold ontology-level canonicalization for noun phrases, as well as
source sentences from which triples are extracted. We also propose metrics for
evaluating each type of canonicalization. On the COMBO dataset, we empirically
compare previously proposed canonicalization methods as well as a few simple
baseline methods based on pretrained language models. We find that properly
encoding the phrases in a triple using pretrained language models results in
better relation canonicalization and ontology-level canonicalization of the
noun phrase. We release our dataset, baselines, and evaluation scripts at
this https URL.

本文提出了一种 Complete Benchmark for Open KG Canonicalization（COMBO）数据集和基于预训练语言模型的基准模型来解决开放知识图谱三元组中主谓宾短语和关系的规范化问题，并且通过提出评估规范化类型的度量标准，证明了使用预训练语言模型对三元组中的每个短语进行嵌入编码可以更好地规范谓语和从属关系。