This paper proposes an approach to improve the runtime efficiency of Japanese
tokenization based on the pointwise linear classification (PLC) framework,
which formulates the whole tokenization process as a sequence of linear
classification problems. Our approach optimizes tokenization by leveraging the
characteristics of the PLC framework and the task definition. Our approach
involves (1) composing multiple classifications into array-based operations,
(2) efficient feature lookup with memory-optimized automata, and (3) three
orthogonal pre-processing methods for reducing actual score calculation. Thus,
our approach makes the tokenization speed 5.7 times faster than the current
approach based on the same model without decreasing tokenization accuracy. Our
implementation is available at this https URL under
the MIT or Apache-2.0 license.

本文提出了一种改进日语分词运行效率的方法，基于点对点线性分类（PLC）框架，通过将整个分词过程化简为一系列线性分类问题，通过利用 PLC 框架和任务定义的特点来优化分词，包括通过组合多个分类为基于数组的操作，通过内存优化的自动机实现高效的特征查找，以及三种正交的预处理方法用于减少实际得分计算，从而使得分词速度比当前基于相同模型的方法提高 5.7 倍而不降低分词准确性。