Central to the Transformer architectures' effectiveness is the self-attention
mechanism, a function that maps queries, keys, and values into a
high-dimensional vector space. However, training the attention weights of
queries, keys, and values is non-trivial from a state of random initialization.
In this paper, we propose two methods. (i) We first address the initialization
problem of Vision Transformers by introducing a simple, yet highly innovative,
initialization approach utilizing Discrete Cosine Transform (DCT) coefficients.
Our proposed DCT-based attention initialization marks a significant gain
compared to traditional initialization strategies; offering a robust foundation
for the attention mechanism. Our experiments reveal that the DCT-based
initialization enhances the accuracy of Vision Transformers in classification
tasks. (ii) We also recognize that since DCT effectively decorrelates image
information in the frequency domain, this decorrelation is useful for
compression because it allows the quantization step to discard many of the
higher-frequency components. Based on this observation, we propose a novel
DCT-based compression technique for the attention function of Vision
Transformers. Since high-frequency DCT coefficients usually correspond to
noise, we truncate the high-frequency DCT components of the input patches. Our
DCT-based compression reduces the size of weight matrices for queries, keys,
and values. While maintaining the same level of accuracy, our DCT compressed
Swin Transformers obtain a considerable decrease in the computational overhead.

Transformer 架构的关键是自注意机制，本文提出两种方法，一种是利用 DCT 系数进行初始化来增强辨别能力，另一种是基于 DCT 的压缩技术来减少计算开销。