Large Language Models have proven highly successful at modelling a variety of
tasks. However, this comes at a steep computational cost that hinders wider
industrial uptake. In this pa005 per, we present MWT: a Multi-Word Tokenizer
that goes beyond word boundaries by representing frequent multi-word
expressions as single tokens. MWTs produce a more compact and efficient
tokenization that yields two benefits: (1) Increase in performance due to a
greater coverage of input data given a fixed sequence length and budget; (2)
Faster and lighter inference due to the ability to reduce the sequence length
with negligible drops in performance. Our results show that MWT is more robust
across shorter sequence lengths, thus allowing for major speedups via early
sequence truncation.

我们提出了一种名为 MWT 的多词标记器，它通过将频繁出现的多词表达式表示为单个标记，从而超越词边界。MWT 产生更紧凑高效的标记化，从而提供两个好处：（1）在固定序列长度和预算的情况下，提高性能，因为能够更全面地覆盖输入数据；（2）通过减少序列长度，使推理速度更快、负载更轻，而性能几乎不受影响。我们的结果表明，MWT 在较短的序列长度上更加稳健，从而通过早期序列截断实现主要的加速。