We consider the problem of aligning two sets of continuous word
representations, corresponding to languages, to a common space in order to
infer a bilingual lexicon. It was recently shown that it is possible to infer
such lexicon, without using any parallel data, by aligning word embeddings
trained on monolingual data. Such line of work is called unsupervised bilingual
induction. By wondering whether it was possible to gain experience in the
progressive learning of several languages, we asked ourselves to what extent we
could integrate the knowledge of a given set of languages when learning a new
one, without having parallel data for the latter. In other words, while keeping
the core problem of unsupervised learning in the latest step, we allowed the
access to other corpora of idioms, hence the name semi-supervised. This led us
to propose a novel formulation, considering the lexicon induction as a ranking
problem for which we used recent tools of this machine learning field. Our
experiments on standard benchmarks, inferring dictionary from English to more
than 20 languages, show that our approach consistently outperforms existing
state of the art benchmark. In addition, we deduce from this new scenario
several relevant conclusions allowing a better understanding of the alignment
phenomenon.

无监督和半监督学习方法在词向量、双语词汇归纳以及排名问题方面取得了重要进展。

半监督学习用于双语词典归纳

Semi-Supervised Learning for Bilingual Lexicon Induction

This paper proposes a new pre-training method, called Code-Switching
Pre-training (CSP for short) for Neural Machine Translation (NMT). Unlike
traditional pre-training method which randomly masks some fragments of the
input sentence, the proposed CSP randomly replaces some words in the source
sentence with their translation words in the target language. Specifically, we
firstly perform lexicon induction with unsupervised word embedding mapping
between the source and target languages, and then randomly replace some words
in the input sentence with their translation words according to the extracted
translation lexicons. CSP adopts the encoder-decoder framework: its encoder
takes the code-mixed sentence as input, and its decoder predicts the replaced
fragment of the input sentence. In this way, CSP is able to pre-train the NMT
model by explicitly making the most of the cross-lingual alignment information
extracted from the source and target monolingual corpus. Additionally, we
relieve the pretrain-finetune discrepancy caused by the artificial symbols like
[mask]. To verify the effectiveness of the proposed method, we conduct
extensive experiments on unsupervised and supervised NMT. Experimental results
show that CSP achieves significant improvements over baselines without
pre-training or with other pre-training methods.

提出了一种新的预训练方法 Code-Switching Pre-training（CSP），它通过利用源语言和目标语言之间的跨语言对齐信息来预先训练神经机器翻译模型，该方法通过无监督的词嵌入映射词典识别源语言和目标语言之间的单词替换，克服了传统模型的随机屏蔽单词的缺点。实验结果表明，该方法在无监督和有监督的机器翻译任务上都取得了显著提高。

神经机器翻译的代码切换预训练

Code-switching pre-training for neural machine translation

Most of recent work in cross-lingual word embeddings is severely
Anglocentric. The vast majority of lexicon induction evaluation dictionaries
are between English and another language, and the English embedding space is
selected by default as the hub when learning in a multilingual setting. With
this work, however, we challenge these practices. First, we show that the
choice of hub language can significantly impact downstream lexicon induction
performance. Second, we both expand the current evaluation dictionary
collection to include all language pairs using triangulation, and also create
new dictionaries for under-represented languages. Evaluating established
methods over all these language pairs sheds light into their suitability and
presents new challenges for the field. Finally, in our analysis we identify
general guidelines for strong cross-lingual embeddings baselines, based on more
than just Anglocentric experiments.

通过研究跨语言词嵌入，本文关注了跨语言词典诱导及其评价指标，并发现了 Anglocentric 实验的局限性，同时提供了适用于所有语言的强大跨语言嵌入的指导方针。

跨语言嵌入是否应以英语为主？

Should All Cross-Lingual Embeddings Speak English?

It has been previously noted that neural machine translation (NMT) is very
sensitive to domain shift. In this paper, we argue that this is a dual effect
of the highly lexicalized nature of NMT, resulting in failure for sentences
with large numbers of unknown words, and lack of supervision for
domain-specific words. To remedy this problem, we propose an unsupervised
adaptation method which fine-tunes a pre-trained out-of-domain NMT model using
a pseudo-in-domain corpus. Specifically, we perform lexicon induction to
extract an in-domain lexicon, and construct a pseudo-parallel in-domain corpus
by performing word-for-word back-translation of monolingual in-domain target
sentences. In five domains over twenty pairwise adaptation settings and two
model architectures, our method achieves consistent improvements without using
any in-domain parallel sentences, improving up to 14 BLEU over unadapted
models, and up to 2 BLEU over strong back-translation baselines.

本文提出了一种无监督的自适应方法，使用伪域内语料库通过微调预先训练的域外 NMT 模型，借助词典诱导提取域内词汇，并通过针对性的单语域内目标句子进行逐词反向翻译构建伪平行域内语料库，在 20 对自适应设置和两种模型架构的五个领域中，我们的方法始终显示出提高，而不需要使用任何域内平行句子，改进了未经过调整的模型高达 14 BLEU 和强反向翻译基线的最高 2 BLEU。