Despite the impressive representation capacity of vision transformer models,
current light-weight vision transformer models still suffer from inconsistent
and incorrect dense predictions at local regions. We suspect that the power of
their self-attention mechanism is limited in shallower and thinner networks. We
propose Lite Vision Transformer (LVT), a novel light-weight transformer network
with two enhanced self-attention mechanisms to improve the model performances
for mobile deployment. For the low-level features, we introduce Convolutional
Self-Attention (CSA). Unlike previous approaches of merging convolution and
self-attention, CSA introduces local self-attention into the convolution within
a kernel of size 3x3 to enrich low-level features in the first stage of LVT.
For the high-level features, we propose Recursive Atrous Self-Attention (RASA),
which utilizes the multi-scale context when calculating the similarity map and
a recursive mechanism to increase the representation capability with marginal
extra parameter cost. The superiority of LVT is demonstrated on ImageNet
recognition, ADE20K semantic segmentation, and COCO panoptic segmentation. The
code is made publicly available.

本研究提出了 Lite Vision Transformer（LVT），一种新型的轻量级 transformer 网络，其中包含两种增强自注意力机制，以提高模型在移动部署中的性能表现，并在 ImageNet 识别、ADE20K 语义分割和 COCO 全景分割中具有优越性能。

具有增强自注意力的轻量级视觉 Transformer

Lite Vision Transformer with Enhanced Self-Attention

Time series forecasting is an important problem across many domains,
including predictions of solar plant energy output, electricity consumption,
and traffic jam situation. In this paper, we propose to tackle such forecasting
problem with Transformer [1]. Although impressed by its performance in our
preliminary study, we found its two major weaknesses: (1) locality-agnostics:
the point-wise dot-product self-attention in canonical Transformer architecture
is insensitive to local context, which can make the model prone to anomalies in
time series; (2) memory bottleneck: space complexity of canonical Transformer
grows quadratically with sequence length $L$, making directly modeling long
time series infeasible. In order to solve these two issues, we first propose
convolutional self-attention by producing queries and keys with causal
convolution so that local context can be better incorporated into attention
mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$
memory cost, improving forecasting accuracy for time series with fine
granularity and strong long-term dependencies under constrained memory budget.
Our experiments on both synthetic data and real-world datasets show that it
compares favorably to the state-of-the-art.

本文提出使用 Transformer 解决时间序列预测问题，针对其点乘自注意力机制的局限性和内存瓶颈问题，先提出了卷积自注意力机制，再通过提出只有 O (L (logL)^2) 内存成本的 LogSparse Transformer 来解决内存瓶颈问题，实验结果表明在预算有限的情况下，其能显著提高对于时间序列的预测准确率。