Since learning is typically very slow in Boltzmann machines, there is a need
to restrict connections within hidden layers. However, the resulting states of
hidden units exhibit statistical dependencies. Based on this observation, we
propose using $l_1/l_2$ regularization upon the activation possibilities of
hidden units in restricted Boltzmann machines to capture the loacal
dependencies among hidden units. This regularization not only encourages hidden
units of many groups to be inactive given observed data but also makes hidden
units within a group compete with each other for modeling observed data. Thus,
the $l_1/l_2$ regularization on RBMs yields sparsity at both the group and the
hidden unit levels. We call RBMs trained with the regularizer \emph{sparse
group} RBMs. The proposed sparse group RBMs are applied to three tasks:
modeling patches of natural images, modeling handwritten digits and pretaining
a deep networks for a classification task. Furthermore, we illustrate the
regularizer can also be applied to deep Boltzmann machines, which lead to
sparse group deep Boltzmann machines. When adapted to the MNIST data set, a
two-layer sparse group Boltzmann machine achieves an error rate of $0.84\%$,
which is, to our knowledge, the best published result on the
permutation-invariant version of the MNIST task.

本文提出了基于 l1/l2 正则化的稀疏组受限玻尔兹曼机，该方法不仅能够在组和隐藏单元级别上实现稀疏性，同时在图像、手写数字和图像分类等任务中取得更好的成绩，并在 MNIST 数据集上实现了 0.84% 的误差率，是目前公布的最佳结果。