Subword tokenization is a common method for vocabulary building in Neural
Machine Translation (NMT) models. However, increasingly complex tasks have
revealed its disadvantages. First, a vocabulary cannot be modified once it is
learned, making it hard to adapt to new words. Second, in multilingual
translation, the imbalance in data volumes across different languages spreads
to the vocabulary, exacerbating translations involving low-resource languages.
While byte-based tokenization addresses these issues, byte-based models
struggle with the low information density inherent in UTF-8 byte sequences.
Previous works enhance token semantics through local contextualization but fail
to select an appropriate contextualizing scope based on the input.
Consequently, we propose the Multi-Scale Contextualization (MSC) method, which
learns contextualized information of varying scales across different hidden
state dimensions. It then leverages the attention module to dynamically
integrate the multi-scale contextualized information. Experiments show that MSC
significantly outperforms subword-based and other byte-based methods in both
multilingual and out-of-domain scenarios. Code can be found in
this https URL

在神经机器翻译（NMT）模型中，子词标记化是一种常见的词汇构建方法，但是越来越复杂的任务使其劣势显现。我们提出了多尺度上下文化（MSC）方法，它通过学习不同隐藏状态维度上的不同尺度的上下文化信息，并利用注意力模块动态地集成多尺度的上下文化信息，以解决这些问题。实验证明，MSC 在多语言和领域外的场景中明显优于子词和其他基于字节的方法。