Transformers achieve unrivalled performance in modelling language, but remain
inefficient in terms of memory and time complexity. A possible remedy is to
reduce the sequence length in the intermediate layers by pooling fixed-length
segments of tokens. Nevertheless, natural units of meaning, such as words or
phrases, display varying sizes. To address this mismatch, we equip language
models with a dynamic-pooling mechanism, which predicts segment boundaries in
an autoregressive fashion. We compare several methods to infer boundaries,
including end-to-end learning through stochastic re-parameterisation,
supervised learning (based on segmentations from subword tokenizers or spikes
in conditional entropy), as well as linguistically motivated boundaries. We
perform character-level evaluation on texts from multiple datasets and
morphologically diverse languages. The results demonstrate that dynamic
pooling, which jointly segments and models language, is often both faster and
more accurate than vanilla Transformers and fixed-length pooling within the
same computational budget.

通过动态 Pooling 和自回归机制，使得 Transformer 模型在其计算资源内的表现更快更准确。