This paper proposes a method to optimize tokenization for the performance
improvement of already trained downstream models. Our method generates
tokenization results attaining lower loss values of a given downstream model on
the training data for restricting vocabularies and trains a tokenizer
reproducing the tokenization results. Therefore, our method can be applied to
variety of tokenization methods, while existing work cannot due to the
simultaneous learning of the tokenizer and the downstream model. This paper
proposes an example of the BiLSTM-based tokenizer with vocabulary restriction,
which can capture wider contextual information for the tokenization process
than non-neural-based tokenization methods used in existing work. Experimental
results on text classification in Japanese, Chinese, and English text
classification tasks show that the proposed method improves performance
compared to the existing methods for tokenization optimization.

本文提出了一种优化已训练后的下游模型性能的分词方法，该方法通过限制词汇量并训练一个生成相应分词结果的分词器，达到更低的下游模型损失值，比现有方法在各项任务中都表现更佳。其中提出的基于 BiLSTM 的分词器能够比现有非神经网络分词方法更好地捕捉上下文信息。