This paper first answers the question "why do the two most powerful techniques Dropout and Batch Normalization (BN) often lead to a worse performance when they are combined together?" in both theoretical and statistical aspects. Theoretically, we find that Dropout would shift the variance of a specific neural unit when we transfer the state of that network from train to test. However, BN would maintain its statistical variance, which is accumulated from the entire learning procedure, in the test phase. The inconsistency of that variance (we name this scheme as "variance shift") causes the unstable numerical behavior in inference that leads to more erroneous predictions finally, when applying Dropout before BN. Thorough experiments on DenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to the uncovered mechanism, we next explore several strategies that modifies Dropout and try to overcome the limitations of their combination by avoiding the variance shift risks.

本文通过理论和统计分析探讨了为何结合 Dropout 和 Batch Normalization 会导致神经网络预测准确率下降的问题，发现该组合会引起特定神经元的方差转移并导致预测不稳定，提出了一些修改 Dropout 的策略来避免方差转移的风险。

通过方差漂移理解Dropout和Batch Normalization之间的不协调性