The design choices in Transformer feed-forward neural networks have resulted
in significant computational and parameter overhead. In this work, we emphasize
the importance of hidden dimension in designing lightweight FFNs, a factor
often overlooked in previous architectures. Guided by this principle, we
introduce PartialFormer, a parameter-efficient Transformer architecture
utilizing multiple smaller FFNs to reduce parameters and computation while
maintaining essential hidden dimensions. These smaller FFNs are integrated into
a multi-head attention system to enable effective collaboration. We also
propose a tailored head scaling strategy to enhance PartialFormer's
capabilities. Furthermore, we present a residual-like attention calculation to
improve depth scaling within PartialFormer. Extensive experiments on 9
translation tasks and 1 abstractive summarization task validate the
effectiveness of our PartialFormer approach. Our code would be available at:
https://github.com/zhengkid/PartialFormer.

本论文介绍了 PartialFormer—— 一种参数有效的 Transformer 架构，利用多个较小的前馈神经网络来减少参数和计算量，同时保持重要的隐藏维度，并通过一个多头注意力系统实现有效的协同工作。还介绍了一种定制的头缩放策略和改进 PartialFormer 深度扩展的注意力计算方法，对 9 个翻译任务和 1 个抽象摘要任务的大量实验验证了 PartialFormer 方法的有效性。