Dense vector representations for textual data are crucial in modern NLP. Word
embeddings and sentence embeddings estimated from raw texts are key in
achieving state-of-the-art results in various tasks requiring semantic
understanding. However, obtaining embeddings at the document level is
challenging due to computational requirements and lack of appropriate data.
Instead, most approaches fall back on computing document embeddings based on
sentence representations. Although there exist architectures and models to
encode documents fully, they are in general limited to English and few other
high-resourced languages. In this work, we provide a systematic comparison of
methods to produce document-level representations from sentences based on
LASER, LaBSE, and Sentence BERT pre-trained multilingual models. We compare
input token number truncation, sentence averaging as well as some simple
windowing and in some cases new augmented and learnable approaches, on 3 multi-
and cross-lingual tasks in 8 languages belonging to 3 different language
families. Our task-based extrinsic evaluations show that, independently of the
language, a clever combination of sentence embeddings is usually better than
encoding the full document as a single unit, even when this is possible. We
demonstrate that while a simple sentence average results in a strong baseline
for classification tasks, more complex combinations are necessary for semantic
tasks.

本研究系统地比较了使用 LASER、LaBSE 和 Sentence BERT 预训练的多语言模型从句子中生成文档级表示的方法，证明了对于多语言任务来说，通常使用句子嵌入的巧妙组合要优于将整个文档编码为单个单元，并且说明了虽然简单的句子平均值对于分类任务来说已经有一定的效果，但对于语义任务来说需要更复杂的组合。

最佳多语言文件嵌入是否只是基于句子嵌入？

Are the Best Multilingual Document Embeddings simply Based on Sentence  Embeddings?

We explore using multilingual document embeddings for nearest neighbor mining
of parallel data. Three document-level representations are investigated: (i)
document embeddings generated by simply averaging multilingual sentence
embeddings; (ii) a neural bag-of-words (BoW) document encoding model; (iii) a
hierarchical multilingual document encoder (HiDE) that builds on our
sentence-level model. The results show document embeddings derived from
sentence-level averaging are surprisingly effective for clean datasets, but
suggest models trained hierarchically at the document-level are more effective
on noisy data. Analysis experiments demonstrate our hierarchical models are
very robust to variations in the underlying sentence embedding quality. Using
document embeddings trained with HiDE achieves state-of-the-art performance on
United Nations (UN) parallel document mining, 94.9% P@1 for en-fr and 97.3% P@1
for en-es.

本研究利用多语言文本嵌入来对平行数据进行最近邻挖掘，通过三种不同的文本层级表示方法，发现用 HiDE 训练产生的文档嵌入结果表现最佳，达到了 UN 平行文档挖掘的最新成果。

层次化文档编码器用于平行语料库挖掘

Hierarchical Document Encoder for Parallel Corpus Mining

In this dissertation we report results of our research on dense distributed
representations of text data. We propose two novel neural models for learning
such representations. The first model learns representations at the document
level, while the second model learns word-level representations.
For document-level representations we propose Binary Paragraph Vector: a
neural network models for learning binary representations of text documents,
which can be used for fast document retrieval. We provide a thorough evaluation
of these models and demonstrate that they outperform the seminal method in the
field in the information retrieval task. We also report strong results in
transfer learning settings, where our models are trained on a generic text
corpus and then used to infer codes for documents from a domain-specific
dataset. In contrast to previously proposed approaches, Binary Paragraph Vector
models learn embeddings directly from raw text data.
For word-level representations we propose Disambiguated Skip-gram: a neural
network model for learning multi-sense word embeddings. Representations learned
by this model can be used in downstream tasks, like part-of-speech tagging or
identification of semantic relations. In the word sense induction task
Disambiguated Skip-gram outperforms state-of-the-art models on three out of
four benchmarks datasets. Our model has an elegant probabilistic
interpretation. Furthermore, unlike previous models of this kind, it is
differentiable with respect to all its parameters and can be trained with
backpropagation. In addition to quantitative results, we present qualitative
evaluation of Disambiguated Skip-gram, including two-dimensional visualisations
of selected word-sense embeddings.

本文研究了稠密分布式文本数据表示的两种新型神经模型：一种用于学习文档级表示，另一种用于学习词级表示。我们提出的模型分别在信息检索和语义关系识别等任务中优于现有方法，并且直接从原始文本数据中学习嵌入。其中，词级表示模型不同于以往的模型，可以用梯度下降算法进行训练。