Significance testing aims to determine whether a proposition about the
population distribution is the truth or not given observations. However,
traditional significance testing often needs to derive the distribution of the
testing statistic, failing to deal with complex nonlinear relationships. In
this paper, we propose to conduct Full Bayesian Significance Testing for neural
networks, called \textit{n}FBST, to overcome the limitation in relationship
characterization of traditional approaches. A Bayesian neural network is
utilized to fit the nonlinear and multi-dimensional relationships with small
errors and avoid hard theoretical derivation by computing the evidence value.
Besides, \textit{n}FBST can test not only global significance but also local
and instance-wise significance, which previous testing methods don't focus on.
Moreover, \textit{n}FBST is a general framework that can be extended based on
the measures selected, such as Grad-\textit{n}FBST, LRP-\textit{n}FBST,
DeepLIFT-\textit{n}FBST, LIME-\textit{n}FBST. A range of experiments on both
simulated and real data are conducted to show the advantages of our method.

提出了一种全贝叶斯显著性测试方法（nFBST），采用贝叶斯神经网络来拟合非线性和多维关系，并通过计算证据值避免了传统方法中的理论派生难题，可以测试全局、局部和实例显著性，并且是一个可根据所选择的度量进行扩展的通用框架。

神经网络的全贝叶斯显著性检验

Full Bayesian Significance Testing for Neural Networks

Significance testing -- especially the paired-permutation test -- has played
a vital role in developing NLP systems to provide confidence that the
difference in performance between two systems (i.e., the test statistic) is not
due to luck. However, practitioners rely on Monte Carlo approximation to
perform this test due to a lack of a suitable exact algorithm. In this paper,
we provide an efficient exact algorithm for the paired-permutation test for a
family of structured test statistics. Our algorithm runs in $\mathcal{O}(GN
(\log GN )(\log N ))$ time where $N$ is the dataset size and $G$ is the range
of the test statistic. We found that our exact algorithm was $10$x faster than
the Monte Carlo approximation with $20000$ samples on a common dataset.

本篇文章提供了一种精确的算法用于针对一类结构化检验统计量的成对置换检验，并指出这种算法要比 Monte Carlo 近似法能够在同一数据集上快 10 倍。

结构化测试统计的精确成对置换检验

Exact Paired-Permutation Testing for Structured Test Statistics

In the sparse linear regression setting, we consider testing the significance
of the predictor variable that enters the current lasso model, in the sequence
of models visited along the lasso solution path. We propose a simple test
statistic based on lasso fitted values, called the covariance test statistic,
and show that when the true model is linear, this statistic has an
$\operatorname {Exp}(1)$ asymptotic distribution under the null hypothesis (the
null being that all truly active variables are contained in the current lasso
model). Our proof of this result for the special case of the first predictor to
enter the model (i.e., testing for a single significant predictor variable
against the global null) requires only weak assumptions on the predictor matrix
$X$. On the other hand, our proof for a general step in the lasso path places
further technical assumptions on $X$ and the generative model, but still allows
for the important high-dimensional case $p>n$, and does not necessarily require
that the current lasso model achieves perfect recovery of the truly active
variables. Of course, for testing the significance of an additional variable
between two nested linear models, one typically uses the chi-squared test,
comparing the drop in residual sum of squares (RSS) to a $\chi^2_1$
distribution. But when this additional variable is not fixed, and has been
chosen adaptively or greedily, this test is no longer appropriate: adaptivity
makes the drop in RSS stochastically much larger than $\chi^2_1$ under the null
hypothesis. Our analysis explicitly accounts for adaptivity, as it must, since
the lasso builds an adaptive sequence of linear models as the tuning parameter
$\lambda$ decreases. In this analysis, shrinkage plays a key role: though
additional variables are chosen adaptively, the coefficients of lasso active
variables are shrunken due to the $\ell_1$ penalty. Therefore, the test
statistic (which is based on lasso fitted values) is in a sense balanced by
these two opposing properties - adaptivity and shrinkage - and its null
distribution is tractable and asymptotically $\operatorname {Exp}(1)$.

本文提出一个基于套索回归路径的协方差检验统计量，结合自适应和收缩的特性，可用于在稀疏线性回归模型下对预测变量重要性进行显著性检验。

Lasso 方法的显著性检验

A significance test for the lasso

Higher criticism, or second-level significance testing, is a
multiple-comparisons concept mentioned in passing by Tukey. It concerns a
situation where there are many independent tests of significance and one is
interested in rejecting the joint null hypothesis. Tukey suggested comparing
the fraction of observed significances at a given \alpha-level to the expected
fraction under the joint null. In fact, he suggested standardizing the
difference of the two quantities and forming a z-score; the resulting z-score
tests the significance of the body of significance tests. We consider a
generalization, where we maximize this z-score over a range of significance
levels 0<\alpha\leq\alpha_0.
We are able to show that the resulting higher criticism statistic is
effective at resolving a very subtle testing problem: testing whether n normal
means are all zero versus the alternative that a small fraction is nonzero. The
subtlety of this ``sparse normal means'' testing problem can be seen from work
of Ingster and Jin, who studied such problems in great detail. In their
studies, they identified an interesting range of cases where the small fraction
of nonzero means is so small that the alternative hypothesis exhibits little
noticeable effect on the distribution of the p-values either for the bulk of
the tests or for the few most highly significant tests.
In this range, when the amplitude of nonzero means is calibrated with the
fraction of nonzero means, the likelihood ratio test for a precisely specified
alternative would still succeed in separating the two hypotheses.

在多重比较中，higher criticism 方法在 testing normal means 是否全部为零的情况下，还有一个小部分是否非零的方差都很小的测试问题中，可以采用最优化 z-score 检验方法来有效地解决此问题，并且当非零均值的振幅与非零均值的分数进行校准时，似然比检验仍然可以成功地分离两个假设。