Self-attention mechanisms are commonly included in a convolutional neural
networks to achieve an improved efficiency performance balance. However, adding
self-attention mechanisms adds additional hyperparameters to tune for the
application at hand. In this work we propose a novel type of DNN analysis
called Multi-Scale Class Representational Response Similarity Analysis
(ClassRepSim) which can be used to identify specific design interventions that
lead to more efficient self-attention convolutional neural network
architectures. Using insights grained from ClassRepSim we propose the Spatial
Transformed Attention Condenser (STAC) module, a novel attention-condenser
based self-attention module. We show that adding STAC modules to ResNet style
architectures can result in up to a 1.6% increase in top-1 accuracy compared to
vanilla ResNet models and up to a 0.5% increase in top-1 accuracy compared to
SENet models on the ImageNet64x64 dataset, at the cost of up to 1.7% increase
in FLOPs and 2x the number of parameters. In addition, we demonstrate that
results from ClassRepSim analysis can be used to select an effective
parameterization of the STAC module resulting in competitive performance
compared to an extensive parameter search.

本文提出了一种多尺度类表示响应相似性分析（ClassRepSim）的 DNN 分析方法，通过该方法提出了一种基于关注凝结器（STAC）的注意力机制模块，将其应用于 ResNet 风格的架构中，实现了在 ImageNet64x64 数据集上比普通 ResNet 模型的 top-1 精度提升了 1.6%、比 SENet 模型的 top-1 精度提升了 0.5% 的结果。同时，还证明了该方法的结果可以用于有效地参数化 STAC 模块，而不需要进行大量的参数搜索。