Transformer-based language models benefit from conditioning on contexts of
hundreds to thousands of previous tokens. What aspects of these contexts
contribute to accurate model prediction? We describe a series of experiments
that measure usable information by selectively ablating lexical and structural
information in transformer language models trained on English Wikipedia. In
both mid- and long-range contexts, we find that several extremely destructive
context manipulations -- including shuffling word order within sentences and
deleting all words other than nouns -- remove less than 15% of the usable
information. Our results suggest that long contexts, but not their detailed
syntactic and propositional content, are important for the low perplexity of
current transformer language models.

通过在 Transformer 语言模型上进行一系列实验，我们发现对于当前 Transformer 语言模型的低困惑度，长上下文至关重要，但上下文的详细句法和命题内容并不重要，且在中长程上下文中，包括重新排列句子中的单词和删除除名词以外的所有单词在内的多种极端破坏性上下文操作仅会移除不到 15％的可用信息。