We propose a method to determine whether a given article was entirely written
by a generative language model versus an alternative situation in which the
article includes some significant edits by a different author, possibly a
human. Our process involves many perplexity tests for the origin of individual
sentences or other text atoms, combining these multiple tests using Higher
Criticism (HC). As a by-product, the method identifies parts suspected to be
edited. The method is motivated by the convergence of the log-perplexity to the
cross-entropy rate and by a statistical model for edited text saying that
sentences are mostly generated by the language model, except perhaps for a few
sentences that might have originated via a different mechanism. We demonstrate
the effectiveness of our method using real data and analyze the factors
affecting its success. This analysis raises several interesting open challenges
whose resolution may improve the method's effectiveness.

我们提出了一种方法，可以确定给定文章是完全由生成式语言模型编写还是在不同作者（可能是人类）的情况下经过了一些重大编辑。我们的方法涉及多个感到困惑的测试来判断单个句子或其他文本单元的来源，并使用 Higher Criticism 方法结合这些多个测试。该方法通过接近对数困惑度与交叉熵率的收敛以及针对编辑文本的统计模型，表明大部分句子都是由语言模型生成的，但也可能有一些句子是通过不同的机制产生的。我们通过使用真实数据展示了我们方法的有效性，并分析了影响其成功的因素。这项分析提出了一些有趣的未解决问题，解决这些问题可能会提高该方法的效果。

利用高阶批判方法分离人工智能生成的文字中的人为因素：一种信息论方法

Separating the Human Touch from AI-Generated Text using Higher  Criticism: An Information-Theoretic Approach

Higher criticism, or second-level significance testing, is a
multiple-comparisons concept mentioned in passing by Tukey. It concerns a
situation where there are many independent tests of significance and one is
interested in rejecting the joint null hypothesis. Tukey suggested comparing
the fraction of observed significances at a given \alpha-level to the expected
fraction under the joint null. In fact, he suggested standardizing the
difference of the two quantities and forming a z-score; the resulting z-score
tests the significance of the body of significance tests. We consider a
generalization, where we maximize this z-score over a range of significance
levels 0<\alpha\leq\alpha_0.
We are able to show that the resulting higher criticism statistic is
effective at resolving a very subtle testing problem: testing whether n normal
means are all zero versus the alternative that a small fraction is nonzero. The
subtlety of this ``sparse normal means'' testing problem can be seen from work
of Ingster and Jin, who studied such problems in great detail. In their
studies, they identified an interesting range of cases where the small fraction
of nonzero means is so small that the alternative hypothesis exhibits little
noticeable effect on the distribution of the p-values either for the bulk of
the tests or for the few most highly significant tests.
In this range, when the amplitude of nonzero means is calibrated with the
fraction of nonzero means, the likelihood ratio test for a precisely specified
alternative would still succeed in separating the two hypotheses.

在多重比较中，higher criticism 方法在 testing normal means 是否全部为零的情况下，还有一个小部分是否非零的方差都很小的测试问题中，可以采用最优化 z-score 检验方法来有效地解决此问题，并且当非零均值的振幅与非零均值的分数进行校准时，似然比检验仍然可以成功地分离两个假设。