Acronym disambiguation means finding the correct meaning of an ambiguous
acronym from the dictionary in a given sentence, which is one of the key points
for scientific document understanding (SDU@AAAI-22). Recently, many attempts
have tried to solve this problem via fine-tuning the pre-trained masked
language models (MLMs) in order to obtain a better acronym representation.
However, the acronym meaning is varied under different contexts, whose
corresponding phrase representation mapped in different directions lacks
discrimination in the entire vector space. Thus, the original representations
of the pre-trained MLMs are not ideal for the acronym disambiguation task. In
this paper, we propose a Simple framework for Contrastive Learning of Acronym
Disambiguation (SimCLAD) method to better understand the acronym meanings.
Specifically, we design a continual contrastive pre-training method that
enhances the pre-trained model's generalization ability by learning the
phrase-level contrastive distributions between true meaning and ambiguous
phrases. The results on the acronym disambiguation of the scientific domain in
English show that the proposed method outperforms all other competitive
state-of-the-art (SOTA) methods.

本文提出了一种用于首字母缩略词消歧的简单对比学习框架 (SimCLAD) 方法，强调了对称性，并通过学习真实含义和歧义短语之间的短语级对比分布来提高预训练模型的泛化能力，并在英文科学文献的首字母缩略词消歧任务上取得了比其他竞争性最先进的方法更好的结果。

SimCLAD: 简单的强化对比学习架构用于缩略语消歧

SimCLAD: A Simple Framework for Contrastive Learning of Acronym  Disambiguation

Previous literatures show that pre-trained masked language models (MLMs) such
as BERT can achieve competitive factual knowledge extraction performance on
some datasets, indicating that MLMs can potentially be a reliable knowledge
source. In this paper, we conduct a rigorous study to explore the underlying
predicting mechanisms of MLMs over different extraction paradigms. By
investigating the behaviors of MLMs, we find that previous decent performance
mainly owes to the biased prompts which overfit dataset artifacts. Furthermore,
incorporating illustrative cases and external contexts improve knowledge
prediction mainly due to entity type guidance and golden answer leakage. Our
findings shed light on the underlying predicting mechanisms of MLMs, and
strongly question the previous conclusion that current MLMs can potentially
serve as reliable factual knowledge bases.

本文对预训练掩码语言模型（MLMs）在不同抽取范式下的预测机制进行了研究，发现之前的良好表现主要归因于偏见提示和外部环境的贡献。实体类型引导和正确答案泄露是知识预测的改进方向。这些发现揭示了 MLMs 的预测机制，同时质疑了现有 MLMs 可以作为可靠事实知识库的结论。

知识性还是教育性猜测？重新审视语言模型作为知识库

Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge  Bases

Adversarial attacks for discrete data (such as texts) have been proved
significantly more challenging than continuous data (such as images) since it
is difficult to generate adversarial samples with gradient-based methods.
Current successful attack methods for texts usually adopt heuristic replacement
strategies on the character or word level, which remains challenging to find
the optimal solution in the massive space of possible combinations of
replacements while preserving semantic consistency and language fluency. In
this paper, we propose \textbf{BERT-Attack}, a high-quality and effective
method to generate adversarial samples using pre-trained masked language models
exemplified by BERT. We turn BERT against its fine-tuned models and other deep
neural models in downstream tasks so that we can successfully mislead the
target models to predict incorrectly. Our method outperforms state-of-the-art
attack strategies in both success rate and perturb percentage, while the
generated adversarial samples are fluent and semantically preserved. Also, the
cost of calculation is low, thus possible for large-scale generations. The code
is available at this https URL

本文提出了使用 BERT 预训练模型的高质量有效的方法 BERT-Attack，用于生成对抗样本并在对下游任务的深度神经模型进行攻击。在成功率和扰动百分比方面，我们的方法优于最先进的攻击策略，而生成的对抗样本则流畅且在语义上保留，且计算成本低，可用于大规模产生。