The recently introduced dropout training criterion for neural networks has
been the subject of much attention due to its simplicity and remarkable
effectiveness as a regularizer, as well as its interpretation as a training
procedure for an exponentially large ensemble of networks that share
parameters. In this work we empirically investigate several questions related
to the efficacy of dropout, specifically as it concerns networks employing the
popular rectified linear activation function. We investigate the quality of the
test time weight-scaling inference procedure by evaluating the geometric
average exactly in small models, as well as compare the performance of the
geometric mean to the arithmetic mean more commonly employed by ensemble
techniques. We explore the effect of tied weights on the ensemble
interpretation by training ensembles of masked networks without tied weights.
Finally, we investigate an alternative criterion based on a biased estimator of
the maximum likelihood ensemble gradient.

研究表明使用 dropout 训练神经网络对于使用流行的修正线性激活函数的网络的有效性有多种问题，探究了测试时权重缩放推断过程的质量，比较了几何平均数和算术平均数的性能，研究了非绑定权重的集合解释对集合的效果，最后，我们调查了基于最大似然集合梯度的带有偏估计器的替代准则。