Modern language models mostly take sub-words as input, a design that balances
the trade-off between vocabulary size, number of parameters, and performance.
However, sub-word tokenization still has disadvantages like not being robust to
noise and difficult to generalize to new languages. Also, the current trend of
scaling up models reveals that larger models require larger embeddings but that
makes parallelization hard. Previous work on image classification proves
splitting raw input into a sequence of chucks is a strong, model-agnostic
inductive bias. Based on this observation, we rethink the existing
character-aware method that takes character-level inputs but makes word-level
sequence modeling and prediction. We overhaul this method by introducing a
cross-attention network that builds word-level representation directly from
bytes, and a sub-word level prediction based on word-level hidden states to
avoid the time and space requirement of word-level prediction. With these two
improvements combined, we have a token free model with slim input embeddings
for downstream tasks. We name our method Byte2Word and perform evaluations on
language modeling and text classification. Experiments show that Byte2Word is
on par with the strong sub-word baseline BERT but only takes up 10\% of
embedding size. We further test our method on synthetic noise and cross-lingual
transfer and find it competitive to baseline methods on both settings.

该论文提出了一种新的方法 Byte2Word，通过引入交叉注意力网络建立单词级别的表示，并基于单词级别的隐藏状态进行子词级别的预测，从而实现了更精简的输入嵌入方式，同时在语言模型和文本分类上表现出与强大的基准模型 BERT 相当的性能。