Vision Transformer (ViT) has shown great potential for various visual tasks
due to its ability to model long-range dependency. However, ViT requires a
large amount of computing resource to compute the global self-attention. In
this work, we propose a ladder self-attention block with multiple branches and
a progressive shift mechanism to develop a light-weight transformer backbone
that requires less computing resources (e.g. a relatively small number of
parameters and FLOPs), termed Progressive Shift Ladder Transformer (PSLT).
First, the ladder self-attention block reduces the computational cost by
modelling local self-attention in each branch. In the meanwhile, the
progressive shift mechanism is proposed to enlarge the receptive field in the
ladder self-attention block by modelling diverse local self-attention for each
branch and interacting among these branches. Second, the input feature of the
ladder self-attention block is split equally along the channel dimension for
each branch, which considerably reduces the computational cost in the ladder
self-attention block (with nearly 1/3 the amount of parameters and FLOPs), and
the outputs of these branches are then collaborated by a pixel-adaptive fusion.
Therefore, the ladder self-attention block with a relatively small number of
parameters and FLOPs is capable of modelling long-range interactions. Based on
the ladder self-attention block, PSLT performs well on several vision tasks,
including image classification, objection detection and person
re-identification. On the ImageNet-1k dataset, PSLT achieves a top-1 accuracy
of 79.9% with 9.2M parameters and 1.9G FLOPs, which is comparable to several
existing models with more than 20M parameters and 4G FLOPs. Code is available
at this https URL

本文介绍了使用阶梯自注意力块和渐进式移位机制开发轻量级转换器骨干 (PSTL)，具有较少的计算资源需求，用于各种视觉任务包括图像分类等，相对于现有的模型，PSTL 具有更少的参数和 FLOPs，但在 ImageNet-1k 数据集上却达到了 79.9％的准确率