State-of-the-art LLMs often rely on scale with high computational costs, which has sparked a research agenda to reduce parameter counts and costs without significantly impacting performance. Our study focuses on Transformer-based LLMs, specifically applying low-rank parametrization to the computationally intensive feedforward networks (FFNs), which are less studied than attention blocks. In contrast to previous works, (i) we explore low-rank parametrization at scale, up to 1.3B parameters; (ii) within Transformer language models rather than convolutional architectures; and (iii) starting from training from scratch. Experiments on the large RefinedWeb dataset show that low-rank parametrization is both efficient (e.g., 2.6$\times$ FFN speed-up with 32\% parameters) and effective during training. Interestingly, these structured FFNs exhibit steeper scaling curves than the original models. Motivated by this finding, we develop the wide and structured networks surpassing the current medium-sized and large-sized Transformer in perplexity and throughput performance.

通过对Transformer-based LLMs中的计算密集型前馈神经网络(FFNs)应用低秩参数化，我们研究表明低秩参数化在尺度上是高效的，能够通过减少参数数量和成本而不显著影响性能，我们开发了宽度和结构化的网络，超过了目前的中型和大型Transformer模型在困惑度和吞吐能力方面的表现。

探究Transformer语言模型中的低秩训练：效率与扩展分析