The development of generative language models that can create long and
coherent textual outputs via autoregression has lead to a proliferation of uses
and a corresponding sweep of analyses as researches work to determine the
limitations of this new paradigm. Unlike humans, these 'Large Language Models'
(LLMs) are highly sensitive to small changes in their inputs, leading to
unwanted inconsistency in their behavior. One problematic inconsistency when
LLMs are used to answer multiple-choice questions or analyze multiple inputs is
order dependency: the output of an LLM can (and often does) change
significantly when sub-sequences are swapped, despite both orderings being
semantically identical. In this paper we present , a technique that guarantees
the output of an LLM will not have order dependence on a specified set of
sub-sequences. We show that this method provably eliminates order dependency,
and that it can be applied to any transformer-based LLM to enable text
generation that is unaffected by re-orderings. Delving into the implications of
our method, we show that, despite our inputs being out of distribution, the
impact on expected accuracy is small, where the expectation is over the order
of uniformly chosen shuffling of the candidate responses, and usually
significantly less in practice. Thus, can be used as a 'dropped-in' method on
fully trained models. Finally, we discuss how our method's success suggests
that other strong guarantees can be obtained on LLM performance via modifying
the input representations.

这篇论文提出了一种技术，可以确保生成式语言模型在给定的子序列集合上不受顺序相关性的影响，从而消除顺序相关性。结果表明，该方法对预期的准确性影响较小，而且可以应用于任何基于 Transformer 的大语料库模型，对文本生成没有影响，进一步讨论了通过修改输入表示可以获得其他强大的 LLM 性能保证的可能性。