We present three innovations in tokenization and subword segmentation. First,
we propose to use unsupervised morphological analysis with Morfessor as
pre-tokenization. Second, we present an algebraic method for obtaining subword
embeddings grounded in a word embedding space. Based on that, we design a novel
subword segmentation algorithm that uses the embeddings, ensuring that the
procedure considers lexical meaning. Third, we introduce an efficient
segmentation algorithm based on a subword bigram model that can be initialized
with the lexically aware segmentation method to avoid using Morfessor and large
embedding tables at inference time. We evaluate the proposed approaches using
two intrinsic metrics and measure their performance on two downstream tasks:
part-of-speech tagging and machine translation. Our experiments show
significant improvements in the morphological plausibility of the segmentation
when evaluated using segmentation precision on morpheme boundaries and improved
R\'enyi efficiency in 8 languages. Although the proposed tokenization methods
do not have a large impact on automatic translation quality, we observe
consistent performance gains in the arguably more morphological task of
part-of-speech tagging.

我们提出了三个在标记化和子词分割中的创新。首先，我们建议使用 Morfessor 的无监督形态分析作为预标记化。其次，我们提出了一种代数方法来获取基于词嵌入空间的子词嵌入。基于此，我们设计了一种使用嵌入的新型子词分割算法，确保该过程考虑了词汇意义。第三，我们引入了一种基于子词二元模型的高效分割算法，可使用词汇感知分割方法进行初始化，以避免在推理时使用 Morfessor 和大型嵌入表。我们使用两个内在指标评估了所提出的方法，并在两个下游任务上测试了其性能：词性标注和机器翻译。我们的实验证明，在各语言上，通过在词素边界上评估的分割精度和改进的 Rényi 效率方面，该分割的形态合理性得到了显著提高。尽管所提出的标记化方法对自动翻译质量影响不大，但我们观察到在词性标注这个更具形态学特征的任务中一直存在性能提升。

词汇基础的子词切分

Lexically Grounded Subword Segmentation

The use of subword embedding has proved to be a major innovation in Neural
Machine Translation (NMT). It helps NMT to learn better context vectors for Low
Resource Languages (LRLs) so as to predict the target words by better modelling
the morphologies of the two languages and also the morphosyntax transfer. Even
so, their performance for translation in Indian language to Indian language
scenario is still not as good as for resource-rich languages. One reason for
this is the relative morphological richness of Indian languages, while another
is that most of them fall into the extremely low resource or zero-shot
categories. Since most major Indian languages use Indic or Brahmi origin
scripts, the text written in them is highly phonetic in nature and phonetically
similar in terms of abstract letters and their arrangements. We use these
characteristics of Indian languages and their scripts to propose an approach
based on common multilingual Latin-based encodings (WX notation) that take
advantage of language similarity while addressing the morphological complexity
issue in NMT. These multilingual Latin-based encodings in NMT, together with
Byte Pair Embedding (BPE) allow us to better exploit their phonetic and
orthographic as well as lexical similarities to improve the translation quality
by projecting different but similar languages on the same orthographic-phonetic
character space. We verify the proposed approach by demonstrating experiments
on similar language pairs (Gujarati-Hindi, Marathi-Hindi, Nepali-Hindi,
Maithili-Hindi, Punjabi-Hindi, and Urdu-Hindi) under low resource conditions.
The proposed approach shows an improvement in a majority of cases, in one case
as much as ~10 BLEU points compared to baseline techniques for similar language
pairs. We also get up to ~1 BLEU points improvement on distant and zero-shot
language pairs.

为了解决印地语交互下，由于语言间巨大的形态学复杂度而导致的低资源语言之间的翻译问题，本文提出了一种基于常见多语种拉丁编码的方法。使用这种方法，结合 Byte Pair Embedding，可以更好地利用印地语相似性，并通过将不同但相似的语言映射到同一正交 - 语音字符空间来改进翻译质量。在低资源条件下，我们验证了提出的方法，并且在大多数情况下得到了改进，在类似语言对（Gujarati-Hindi，Marathi-Hindi，Nepali-Hindi，Maithili - Hindi，Punjabi-Hindi 和 Urdu-Hindi）中，在一个案例中提高了约 10 BLEU 分数，而在远程和零 - shot 语言对中还提高了约 1 个 BLEU 分数。