State-of-the-art results in large language models (LLMs) often rely on scale,
which becomes computationally expensive. This has sparked a research agenda to
reduce these models' parameter count and computational costs without
significantly impacting their performance. Our study focuses on
transformer-based LLMs, specifically targeting the computationally intensive
feedforward networks (FFN), which are less studied than attention blocks. We
consider three candidate linear layer approximations in the FFN by combining
efficient low-rank and block-diagonal matrices. In contrast to many previous
works that examined these approximations, our study i) explores these
structures from the training-from-scratch perspective, ii) scales up to 1.3B
parameters, and iii) is conducted within recent Transformer-based LLMs rather
than convolutional architectures. We first demonstrate they can lead to actual
computational gains in various scenarios, including online decoding when using
a pre-merge technique. Additionally, we propose a novel training regime, called
\textit{self-guided training}, aimed at improving the poor training dynamics
that these approximations exhibit when used from initialization. Experiments on
the large RefinedWeb dataset show that our methods are both efficient and
effective for training and inference. Interestingly, these structured FFNs
exhibit steeper scaling curves than the original models. Further applying
self-guided training to the structured matrices with 32\% FFN parameters and
2.5$\times$ speed-up enables only a 0.4 perplexity increase under the same
training FLOPs. Finally, we develop the wide and structured networks surpassing
the current medium-sized and large-sized Transformer in perplexity and
throughput performance. Our code is available at
https://github.com/CLAIRE-Labo/StructuredFFN/tree/main.

通过结构化的前馈网络，使用线性层近似来减少大型语言模型的参数数量和计算成本，同时提出一种自我引导训练的方法来改善这种近似所表现出的训练动力学，从而在训练和推断中实现高效和有效的结果。