While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity-inducing beta-Bernoulli prior. It allows each neuron to be evolved either to be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsity- inducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta-Bernoulli dropout on multiple public datasets, on which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.

我们提出了一种自适应的变分dropout方法，在此方法中使每个神经元进化为通用或特定于某些输入，或完全被删除，以允许网络耐受更大程度的稀疏性，同时消除功能中冗余项，我们在多个公共数据集上验证了我们的依赖性变分beta-Bernoulli dropout方法，并获得比基准方法更紧凑的网络，其中在基本网络上保持准确性。

依赖变分Beta-Bernoulli Dropout的自适应网络稀疏化