In this work, we address two main shortcomings of transformer architectures: input corruption and rank collapse in their output representation. We unveil self-attention as an autonomous state-space model that inherently promotes smoothness in its solutions, leading to lower-rank outputs and diminished representation capacity. Moreover, the steady-state solution of the model is sensitive to input perturbations. We incorporate a Proportional-Integral-Derivative (PID) closed-loop feedback control system with a reference point into the model to improve robustness and representation capacity. This integration aims to preserve high-frequency details while bolstering model stability, rendering it more noise-resilient. The resulting controlled state-space model is theoretically proven robust and adept at addressing the rank collapse. Motivated by this control framework, we derive a novel class of transformers, PID-controlled Transformer (PIDformer), aimed at improving robustness and mitigating the rank-collapse issue inherent in softmax transformers. We empirically evaluate the model for advantages and robustness against baseline transformers across various practical tasks, including object classification, image segmentation, and language modeling.

我们提出了一种改进transformer架构的方法，通过引入自我注意力作为自主状态空间模型，控制系统对输入的干扰进行反馈控制，提高模型的稳定性和容错能力，从而解决了softmax transformer中的秩坍缩问题。我们通过在目标分类、图像分割和语言建模等实际任务中的实验证明了该模型的优势和鲁棒性。

PIDformer: Transformer遇见控制理论