Understanding the representational power of Restricted Boltzmann Machines
(RBMs) with multiple layers is an ill-understood problem and is an area of
active research. Motivated from the approach of \emph{Inherent Structure
formalism} (Stillinger & Weber, 1982), extensively used in analysing Spin
Glasses, we propose a novel measure called \emph{Inherent Structure Capacity}
(ISC), which characterizes the representation capacity of a fixed architecture
RBM by the expected number of modes of distributions emanating from the RBM
with parameters drawn from a prior distribution. Though ISC is intractable, we
show that for a single layer RBM architecture ISC approaches a finite constant
as number of hidden units are increased and to further improve the ISC, one
needs to add a second layer. Furthermore, we introduce \emph{Lean} RBMs, which
are multi-layer RBMs where each layer can have at-most $O(n)$ units with the
number of visible units being n. We show that for every single layer RBM with
$\Omega(n^{2+r}), r \ge 0$, hidden units there exists a two-layered \emph{lean}
RBM with $\Theta(n^2)$ parameters with the same ISC, establishing that 2 layer
RBMs can achieve the same representational power as single-layer RBMs but using
far fewer number of parameters. To the best of our knowledge, this is the first
result which quantitatively establishes the need for layering.

通过引入内在结构容量 (Inherent Structure Capacity，ISC) 这一新量来衡量 RBM 的表示能力，我们证明当使用一个或两层 RBM 时，ISC 随着 RBM 中隐藏单元的数量增加而趋于一个有限常数，而两层 RBM 与单层 RBM 具有相同的表示能力，但使用的参数更少。