We present three innovations in tokenization and subword segmentation. First,
we propose to use unsupervised morphological analysis with Morfessor as
pre-tokenization. Second, we present an algebraic method for obtaining subword
embeddings grounded in a word embedding space. Based on that, we design a novel
subword segmentation algorithm that uses the embeddings, ensuring that the
procedure considers lexical meaning. Third, we introduce an efficient
segmentation algorithm based on a subword bigram model that can be initialized
with the lexically aware segmentation method to avoid using Morfessor and large
embedding tables at inference time. We evaluate the proposed approaches using
two intrinsic metrics and measure their performance on two downstream tasks:
part-of-speech tagging and machine translation. Our experiments show
significant improvements in the morphological plausibility of the segmentation
when evaluated using segmentation precision on morpheme boundaries and improved
R\'enyi efficiency in 8 languages. Although the proposed tokenization methods
do not have a large impact on automatic translation quality, we observe
consistent performance gains in the arguably more morphological task of
part-of-speech tagging.

我们提出了三个在标记化和子词分割中的创新。首先，我们建议使用 Morfessor 的无监督形态分析作为预标记化。其次，我们提出了一种代数方法来获取基于词嵌入空间的子词嵌入。基于此，我们设计了一种使用嵌入的新型子词分割算法，确保该过程考虑了词汇意义。第三，我们引入了一种基于子词二元模型的高效分割算法，可使用词汇感知分割方法进行初始化，以避免在推理时使用 Morfessor 和大型嵌入表。我们使用两个内在指标评估了所提出的方法，并在两个下游任务上测试了其性能：词性标注和机器翻译。我们的实验证明，在各语言上，通过在词素边界上评估的分割精度和改进的 Rényi 效率方面，该分割的形态合理性得到了显著提高。尽管所提出的标记化方法对自动翻译质量影响不大，但我们观察到在词性标注这个更具形态学特征的任务中一直存在性能提升。