More transformer blocks with residual connections have recently achieved
impressive results on various tasks. To achieve better performance with fewer
trainable parameters, recent methods are proposed to go shallower by parameter
sharing or model compressing along with the depth. However, weak modeling
capacity limits their performance. Contrastively, going wider by inducing more
trainable matrixes and parameters would produce a huge model requiring advanced
parallelism to train and inference.
In this paper, we propose a parameter-efficient framework, going wider
instead of deeper. Specially, following existing works, we adapt parameter
sharing to compress along depth. But, such deployment would limit the
performance. To maximize modeling capacity, we scale along model width by
replacing feed-forward network (FFN) with mixture-of-experts (MoE). Across
transformer blocks, instead of sharing normalization layers, we propose to use
individual layernorms to transform various semantic representations in a more
parameter-efficient way. To evaluate our plug-and-run framework, we design
WideNet and conduct comprehensive experiments on popular computer vision and
natural language processing benchmarks. On ImageNet-1K, our best model
outperforms Vision Transformer (ViT) by $1.5\%$ with $0.72 \times$ trainable
parameters. Using $0.46 \times$ and $0.13 \times$ parameters, our WideNet can
still surpass ViT and ViT-MoE by $0.8\%$ and $2.1\%$, respectively. On four
natural language processing datasets, WideNet outperforms ALBERT by $1.8\%$ on
average and surpass BERT using factorized embedding parameterization by $0.8\%$
with fewer parameters.

本文提出了一种宽架构、混合专家 (MoE) 替代前馈神经网络 (FFN) 的参数高效框架，通过参数共享以压缩深度，并使用独立的 layernorms 来转换各种语义表示，实验结果在多个计算机视觉和自然语言处理基准测试中表现优异，最佳模型能以 0.72 倍的可训练参数超越 Vision Transformer 1.5％、超越 ALBERT 1.8％平均表现和使用分解嵌入参数化的 BERT 0.8％。