Subword tokenization is a commonly used input pre-processing step in most
recent NLP models. However, it limits the models' ability to leverage
end-to-end task learning. Its frequency-based vocabulary creation compromises
tokenization in low-resource languages, leading models to produce suboptimal
representations. Additionally, the dependency on a fixed vocabulary limits the
subword models' adaptability across languages and domains. In this work, we
propose a vocabulary-free neural tokenizer by distilling segmentation
information from heuristic-based subword tokenization. We pre-train our
character-based tokenizer by processing unique words from multilingual corpus,
thereby extensively increasing word diversity across languages. Unlike the
predefined and fixed vocabularies in subword methods, our tokenizer allows
end-to-end task learning, resulting in optimal task-specific tokenization. The
experimental results show that replacing the subword tokenizer with our neural
tokenizer consistently improves performance on multilingual (NLI) and
code-switching (sentiment analysis) tasks, with larger gains in low-resource
languages. Additionally, our neural tokenizer exhibits a robust performance on
downstream tasks when adversarial noise is present (typos and misspelling),
further increasing the initial improvements over statistical subword
tokenizers.

本文提出了一种无需词汇表的神经分词器，通过处理多语言语料库中的独特单词来预训练基于字符的分词器，从而广泛增加语言间的单词多样性，进而克服了子词分词存在的一些问题，如无法进行端到端任务学习，适应性不足，以及在低资源语言中表现欠佳等，实验证明该神经分词器能显著提升多语言（NLI）和混合语言情感分析等任务的性能并且具有较强的鲁棒性。