Current state-of-the-art models for natural language understanding require a
preprocessing step to convert raw text into discrete tokens. This process known
as tokenization relies on a pre-built vocabulary of words or sub-word
morphemes. This fixed vocabulary limits the model's robustness to spelling
errors and its capacity to adapt to new domains. In this work, we introduce a
novel open-vocabulary language model that adopts a hierarchical two-level
approach: one at the word level and another at the sequence level. Concretely,
we design an intra-word module that uses a shallow Transformer architecture to
learn word representations from their characters, and a deep inter-word
Transformer module that contextualizes each word representation by attending to
the entire word sequence. Our model thus directly operates on character
sequences with explicit awareness of word boundaries, but without biased
sub-word or word-level vocabulary. Experiments on various downstream tasks show
that our method outperforms strong baselines. We also demonstrate that our
hierarchical model is robust to textual corruption and domain shift.

该研究引入了一种新颖的开放词汇语言模型，包含两个层次：词级和序列级，并通过对字符的表示以及全局的序列级别上下文调整，使模型直接处理字符序列，而不是子词或词级别的词汇表，取得了比强基准表现更好的效果，并且具有文本破坏和域移位的鲁棒性。