One of the core pillars of efficient deep learning methods is architectural improvements such as the residual/skip connection, which has led to significantly better model convergence and quality. Since then the residual connection has become ubiquitous in not just convolutional neural networks but also transformer-based architectures, the backbone of LLMs. In this paper we introduce \emph{Learned Augmented Residual Layer} (LAuReL) -- a novel generalization of the canonical residual connection -- with the goal to be an in-situ replacement of the latter while outperforming on both model quality and footprint metrics. Our experiments show that using \laurel can help boost performance for both vision and language models. For example, on the ResNet-50, ImageNet 1K task, it achieves $60\%$ of the gains from adding an extra layer, while only adding $0.003\%$ more parameters, and matches it while adding $2.6\times$ fewer parameters.

本研究针对深度学习模型中残差连接的效率和性能提升问题，提出了一种新的方法——学习增强残差层（LAuReL），旨在作为传统残差连接的替代，实现更优的模型质量和占用率。实验结果表明，LAuReL在视觉和语言模型上均表现出显著的性能提升，其在ResNet-50上的表现几乎等同于增加一个额外层的效果，但参数增加极小，显示出其在深度学习架构中的巨大潜力。

LAuReL：学习增强残差层