Multi-layer perceptron (MLP) is a fundamental component of deep learning that
has been extensively employed for various problems. However, recent empirical
successes in MLP-based architectures, particularly the progress of the
MLP-Mixer, have revealed that there is still hidden potential in improving MLPs
to achieve better performance. In this study, we reveal that the MLP-Mixer
works effectively as a wide MLP with certain sparse weights. Initially, we
clarify that the mixing layer of the Mixer has an effective expression as a
wider MLP whose weights are sparse and represented by the Kronecker product.
This expression naturally defines a permuted-Kronecker (PK) family, which can
be regarded as a general class of mixing layers and is also regarded as an
approximation of Monarch matrices. Subsequently, because the PK family
effectively constitutes a wide MLP with sparse weights, one can apply the
hypothesis proposed by Golubeva, Neyshabur and Gur-Ari (2021) that the
prediction performance improves as the width (sparsity) increases when the
number of weights is fixed. We empirically verify this hypothesis by maximizing
the effective width of the MLP-Mixer, which enables us to determine the
appropriate size of the mixing layers quantitatively.

本文研究了多层感知机，重点探究了其深化效果和改进方法。其中提出了 MLP-Mixer 与宽 MLP 具有稀疏权重的相关性，并从预测性能优化的角度进行了实验验证。

MLP-Mixer 作为一个宽且稀疏的多层感知机

MLP-Mixer as a Wide and Sparse MLP

We propose an adaptive projection-gradient descent-shrinkage-splitting method
(APGDSSM) to integrate penalty based channel pruning into quantization-aware
training (QAT). APGDSSM concurrently searches weights in both the quantized
subspace and the sparse subspace. APGDSSM uses shrinkage operator and a
splitting technique to create sparse weights, as well as the Group Lasso
penalty to push the weight sparsity into channel sparsity. In addition, we
propose a novel complementary transformed l1 penalty to stabilize the training
for extreme compression.

提出了一种自适应投影 - 梯度下降收缩分割方法 (APGDSSM)，该方法将基于惩罚的通道剪枝集成到量化感知训练 (QAT) 中。APGDSSM 同时搜索量化子空间和稀疏子空间中的权重，并使用收缩算子和分割技术创建稀疏权重，以及使用 Group Lasso 惩罚将权重稀疏性推入通道稀疏性。此外，提出了一种新颖的补充变换 L1 惩罚方法以稳定极端压缩训练。