We present a subword regularization method for WordPiece, which uses a
maximum matching algorithm for tokenization. The proposed method,
MaxMatch-Dropout, randomly drops words in a search using the maximum matching
algorithm. It realizes finetuning with subword regularization for popular
pretrained language models such as BERT-base. The experimental results
demonstrate that MaxMatch-Dropout improves the performance of text
classification and machine translation tasks as well as other subword
regularization methods. Moreover, we provide a comparative analysis of subword
regularization methods: subword regularization with SentencePiece (Unigram),
BPE-Dropout, and MaxMatch-Dropout.

本文提出了一种用于 WordPiece 的子词规则化方法，使用最大匹配算法进行分词，并通过 MaxMatch-Dropout 方法实现对 BERT-base 等预训练语言模型的微调，实验证明该方法在文本分类和机器翻译任务以及其他子词规则化方法中的性能表现明显优于其他方法，同时还进行了 SentencePiece (Unigram)、BPE-Dropout 和 MaxMatch-Dropout 规则化方法的比较分析。

MaxMatch-Dropout: WordPiece 的子词规范化

MaxMatch-Dropout: Subword Regularization for WordPiece

Tokenization is a fundamental preprocessing step for almost all NLP tasks. In
this paper, we propose efficient algorithms for the WordPiece tokenization used
in BERT, from single-word tokenization to general text (e.g., sentence)
tokenization. When tokenizing a single word, WordPiece uses a
longest-match-first strategy, known as maximum matching. The best known
algorithms so far are O(n^2) (where n is the input length) or O(nm) (where m is
the maximum vocabulary token length). We propose a novel algorithm whose
tokenization complexity is strictly O(n). Our method is inspired by the
Aho-Corasick algorithm. We introduce additional linkages on top of the trie
built from the vocabulary, allowing smart transitions when the trie matching
cannot continue. For general text, we further propose an algorithm that
combines pre-tokenization (splitting the text into words) and our linear-time
WordPiece method into a single pass. Experimental results show that our method
is 8.2x faster than HuggingFace Tokenizers and 5.1x faster than TensorFlow Text
on average for general text tokenization.

本文提出 WordPiece 的高效算法以及针对单词和一般文本的标记化方法，包括将预标记文本和线性时间算法相结合，此方法相对于现有方法可提高 8.2 倍到 5.1 倍的效率。