Continuous word representation (aka word embedding) is a basic building block
in many neural network-based models used in natural language processing tasks.
Although it is widely accepted that words with similar semantics should be
close to each other in the embedding space, we find that word embeddings
learned in several tasks are biased towards word frequency: the embeddings of
high-frequency and low-frequency words lie in different subregions of the
embedding space, and the embedding of a rare word and a popular word can be far
from each other even if they are semantically similar. This makes learned word
embeddings ineffective, especially for rare words, and consequently limits the
performance of these neural network models. In this paper, we develop a neat,
simple yet effective way to learn \emph{FRequency-AGnostic word Embedding}
(FRAGE) using adversarial training. We conducted comprehensive studies on ten
datasets across four natural language processing tasks, including word
similarity, language modeling, machine translation and text classification.
Results show that with FRAGE, we achieve higher performance than the baselines
in all tasks.

本文介绍了一种通过对抗性训练学习频率不可知词嵌入（FRAGE）的方法，可使得语义相似的高频和低频词语在同一词嵌入空间中，实验表明此方法在十种数据集上均优于基准。

FRAGE: 频率无关字表示

FRAGE: Frequency-Agnostic Word Representation

Unsupervised learning of syntactic structure is typically performed using
generative models with discrete latent variables and multinomial parameters. In
most cases, these models have not leveraged continuous word representations. In
this work, we propose a novel generative model that jointly learns discrete
syntactic structure and continuous word representations in an unsupervised
fashion by cascading an invertible neural network with a structured generative
prior. We show that the invertibility condition allows for efficient exact
inference and marginal likelihood computation in our model so long as the prior
is well-behaved. In experiments we instantiate our approach with both Markov
and tree-structured priors, evaluating on two tasks: part-of-speech (POS)
induction, and unsupervised dependency parsing without gold POS annotation. On
the Penn Treebank, our Markov-structured model surpasses state-of-the-art
results on POS induction. Similarly, we find that our tree-structured model
achieves state-of-the-art performance on unsupervised dependency parsing for
the difficult training condition where neither gold POS annotation nor
punctuation-based constraints are available.

提出了一种新颖的生成模型，通过串联一个可逆神经网络和一个结构化的生成先验，在无监督的情况下联合学习离散的句法结构和连续的词表示，实现了高效的精确推断和边缘似然计算，并在两个任务中进行了实验评估。