Pre-trained contextual language models are ubiquitously employed for language
understanding tasks, but are unsuitable for resource-constrained systems.
Noncontextual word embeddings are an efficient alternative in these settings.
Such methods typically use one vector to encode multiple different meanings of
a word, and incur errors due to polysemy. This paper proposes a two-stage
method to distill multiple word senses from a pre-trained language model (BERT)
by using attention over the senses of a word in a context and transferring this
sense information to fit multi-sense embeddings in a skip-gram-like framework.
We demonstrate an effective approach to training the sense disambiguation
mechanism in our model with a distribution over word senses extracted from the
output layer embeddings of BERT. Experiments on the contextual word similarity
and sense induction tasks show that this method is superior to or competitive
with state-of-the-art multi-sense embeddings on multiple benchmark data sets,
and experiments with an embedding-based topic model (ETM) demonstrates the
benefits of using this multi-sense embedding in a downstream application.

该研究提出了一种两阶段的方法，使用上下文中单词意义的注意力来提取一个单词的多重意义，并将其传递到 skip-gram 模型中，以生成多重意义的嵌入，这比现有技术表现更好，可应用于主题建模。

使用 BERT 知识蒸馏进行词义归纳

Word Sense Induction with Knowledge Distillation from BERT

Multilingual models have demonstrated impressive cross-lingual transfer
performance. However, test sets like XNLI are monolingual at the example level.
In multilingual communities, it is common for polyglots to code-mix when
conversing with each other. Inspired by this phenomenon, we present two strong
black-box adversarial attacks (one word-level, one phrase-level) for
multilingual models that push their ability to handle code-mixed sentences to
the limit. The former uses bilingual dictionaries to propose perturbations and
translations of the clean example for sense disambiguation. The latter directly
aligns the clean example with its translations before extracting phrases as
perturbations. Our phrase-level attack has a success rate of 89.75% against
XLM-R-large, bringing its average accuracy of 79.85 down to 8.18 on XNLI.
Finally, we propose an efficient adversarial training scheme that trains in the
same number of steps as the original model and show that it improves model
accuracy.

本文介绍两种针对多语言模型的黑盒对抗攻击和一种高效的对抗性训练方案，以提高多语言模型处理混合语言的表现。短语级攻击的成功率为 89.75％，将 XLM-R-large 的平均准确性从 79.85 降至 8.18。

芝麻街上的混合语言：对抗性多语种者的黎明

Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots

Prepositions are very common and very ambiguous, and understanding their
sense is critical for understanding the meaning of the sentence. Supervised
corpora for the preposition-sense disambiguation task are small, suggesting a
semi-supervised approach to the task. We show that signals from unannotated
multilingual data can be used to improve supervised preposition-sense
disambiguation. Our approach pre-trains an LSTM encoder for predicting the
translation of a preposition, and then incorporates the pre-trained encoder as
a component in a supervised classification system, and fine-tunes it for the
task. The multilingual signals consistently improve results on two
preposition-sense datasets.

本研究采用半监督方法来增强介词意义的自动消歧功能。通过在未标注的多语数据上进行预训练，利用一个 LSTM 编码器来预测介词的翻译，然后在一个监督分类系统中将其作为组成部分进行微调，结果表明这种多语信号能够显著提高两个介词意义数据集的结果表现。