Balancing model complexity against the information contained in observed data
is the central challenge to learning. In order for complexity-efficient models
to exist and be discoverable in high dimensions, we require a computational
framework that relates a credible notion of complexity to simple parameter
representations. Further, this framework must allow excess complexity to be
gradually removed via gradient-based optimization. Our n-ary, or n-argument,
activation functions fill this gap by approximating belief functions
(probabilistic Boolean logic) using logit representations of probability. Just
as Boolean logic determines the truth of a consequent claim from relationships
among a set of antecedent propositions, probabilistic formulations generalize
predictions when antecedents, truth tables, and consequents all retain
uncertainty. Our activation functions demonstrate the ability to learn
arbitrary logic, such as the binary exclusive disjunction (p xor q) and ternary
conditioned disjunction ( c ? p : q ), in a single layer using an activation
function of matching or greater arity. Further, we represent belief tables
using a basis that directly associates the number of nonzero parameters to the
effective arity of the belief function, thus capturing a concrete relationship
between logical complexity and efficient parameter representations. This opens
optimization approaches to reduce logical complexity by inducing parameter
sparsity.

研究文章探讨了如何使用 n-ary 激活函数来填补高维模型中的可信复杂性和效率计算框架之间的空白，通过梯度优化来逐渐消除过剩复杂度，并使用零参数相关性表示对置信表进行建模和推断，从而加速对参数而非像素的优化。

自适应 n 元激活函数用于概率布尔逻辑

Adaptive n-ary Activation Functions for Probabilistic Boolean Logic

Sparse Principal Component Analysis (PCA) is a dimensionality reduction
technique wherein one seeks a low-rank representation of a data matrix with
additional sparsity constraints on the obtained representation. We consider two
probabilistic formulations of sparse PCA: a spiked Wigner and spiked Wishart
(or spiked covariance) model. We analyze an Approximate Message Passing (AMP)
algorithm to estimate the underlying signal and show, in the high dimensional
limit, that the AMP estimates are information-theoretically optimal. As an
immediate corollary, our results demonstrate that the posterior expectation of
the underlying signal, which is often intractable to compute, can be obtained
using a polynomial-time scheme. Our results also effectively provide a
single-letter characterization of the sparse PCA problem.

本文讨论了两种概率稀疏主成分分析模型：钉住 Wigner 模型和钉住 Wishart 模型，并分析了一个用于估计基本信号的近似信息传递 (AMP) 算法。在高维极限下，AMP 估计是信息理论上的最优。此外，本文提供了稀疏 PCA 问题的单字母特征。