We consider the capacity of \emph{treelike committee machines} (TCM) neural
networks. Relying on Random Duality Theory (RDT), \cite{Stojnictcmspnncaprdt23}
recently introduced a generic framework for their capacity analysis. An upgrade
based on the so-called \emph{partially lifted} RDT (pl RDT) was then presented
in \cite{Stojnictcmspnncapliftedrdt23}. Both lines of work focused on the
networks with the most typical, \emph{sign}, activations. Here, on the other
hand, we focus on networks with other, more general, types of activations and
show that the frameworks of
\cite{Stojnictcmspnncaprdt23,Stojnictcmspnncapliftedrdt23} are sufficiently
powerful to enable handling of such scenarios as well. In addition to the
standard \emph{linear} activations, we uncover that particularly convenient
results can be obtained for two very commonly used activations, namely, the
\emph{quadratic} and \emph{rectified linear unit (ReLU)} ones. In more concrete
terms, for each of these activations, we obtain both the RDT and pl RDT based
memory capacities upper bound characterization for \emph{any} given (even)
number of the hidden layer neurons, $d$. In the process, we also uncover the
following two, rather remarkable, facts: 1) contrary to the common wisdom, both
sets of results show that the bounding capacity decreases for large $d$ (the
width of the hidden layer) while converging to a constant value; and 2) the
maximum bounding capacity is achieved for the networks with precisely
\textbf{\emph{two}} hidden layer neurons! Moreover, the large $d$ converging
values are observed to be in excellent agrement with the statistical physics
replica theory based predictions.

根据随机二重性理论 (RDT)，本研究利用部分整体化 RDT 的框架分析了具有不同激活函数的树状委员会机器 (TCM) 神经网络的容量，特别关注于线性、二次和修正线性单元 (ReLU) 激活函数，并发现隐藏层神经元个数为两个时容量最大。

固定宽度树状神经网络容量分析 -- 通用激活函数

Fixed width treelike neural networks capacity analysis -- generic  activations

Recent progress in studying \emph{treelike committee machines} (TCM) neural
networks (NN) in
\cite{Stojnictcmspnncaprdt23,Stojnictcmspnncapliftedrdt23,Stojnictcmspnncapdiffactrdt23}
showed that the Random Duality Theory (RDT) and its a \emph{partially
lifted}(pl RDT) variant are powerful tools that can be used for very precise
networks capacity analysis. Here, we consider \emph{wide} hidden layer networks
and uncover that certain aspects of numerical difficulties faced in
\cite{Stojnictcmspnncapdiffactrdt23} miraculously disappear. In particular, we
employ recently developed \emph{fully lifted} (fl) RDT to characterize the
\emph{wide} ($d\rightarrow \infty$) TCM nets capacity. We obtain explicit,
closed form, capacity characterizations for a very generic class of the hidden
layer activations. While the utilized approach significantly lowers the amount
of the needed numerical evaluations, the ultimate fl RDT usefulness and success
still require a solid portion of the residual numerical work. To get the
concrete capacity values, we take four very famous activations examples:
\emph{\textbf{ReLU}}, \textbf{\emph{quadratic}}, \textbf{\emph{erf}}, and
\textbf{\emph{tanh}}. After successfully conducting all the residual numerical
work for all of them, we uncover that the whole lifting mechanism exhibits a
remarkably rapid convergence with the relative improvements no better than
$\sim 0.1\%$ happening already on the 3-rd level of lifting. As a convenient
bonus, we also uncover that the capacity characterizations obtained on the
first and second level of lifting precisely match those obtained through the
statistical physics replica theory methods in \cite{ZavPeh21} for the generic
and in \cite{BalMalZech19} for the ReLU activations.

研究表明，树状委员会机器（TCM）神经网络和 Random Duality Theory（RDT）及其部分提升（pl RDT）变体是用于精确网络容量分析的强大工具，而全面提升（fl）RDT 可用于描述宽（d→∞）TCM 网络的容量，通过对多种激活函数进行数值计算，发现整个提升机制的收敛速度非常快，而第一和第二级提升的容量表征与统计物理复制方法获得的结果相吻合。