In this work, we address two main shortcomings of transformer architectures:
input corruption and rank collapse in their output representation. We unveil
self-attention as an autonomous state-space model that inherently promotes
smoothness in its solutions, leading to lower-rank outputs and diminished
representation capacity. Moreover, the steady-state solution of the model is
sensitive to input perturbations. We incorporate a
Proportional-Integral-Derivative (PID) closed-loop feedback control system with
a reference point into the model to improve robustness and representation
capacity. This integration aims to preserve high-frequency details while
bolstering model stability, rendering it more noise-resilient. The resulting
controlled state-space model is theoretically proven robust and adept at
addressing the rank collapse. Motivated by this control framework, we derive a
novel class of transformers, PID-controlled Transformer (PIDformer), aimed at
improving robustness and mitigating the rank-collapse issue inherent in softmax
transformers. We empirically evaluate the model for advantages and robustness
against baseline transformers across various practical tasks, including object
classification, image segmentation, and language modeling.

我们提出了一种改进 transformer 架构的方法，通过引入自我注意力作为自主状态空间模型，控制系统对输入的干扰进行反馈控制，提高模型的稳定性和容错能力，从而解决了 softmax transformer 中的秩坍缩问题。我们通过在目标分类、图像分割和语言建模等实际任务中的实验证明了该模型的优势和鲁棒性。