Visual perception tasks are predominantly solved by Vision Transformer (ViT)
architectures, which, despite their effectiveness, encounter a computational
bottleneck due to the quadratic complexity of computing self-attention. This
inefficiency is largely due to the self-attention heads capturing redundant
token interactions, reflecting inherent redundancy within visual data. Many
works have aimed to reduce the computational complexity of self-attention in
ViTs, leading to the development of efficient and sparse transformer
architectures. In this paper, viewing through the efficiency lens, we realized
that introducing any sparse self-attention strategy in ViTs can keep the
computational overhead low. However, these strategies are sub-optimal as they
often fail to capture fine-grained visual details. This observation leads us to
propose a general, efficient, sparse architecture, named Fibottention, for
approximating self-attention with superlinear complexity that is built upon
Fibonacci sequences. The key strategies in Fibottention include: it excludes
proximate tokens to reduce redundancy, employs structured sparsity by design to
decrease computational demands, and incorporates inception-like diversity
across attention heads. This diversity ensures the capture of complementary
information through non-overlapping token interactions, optimizing both
performance and resource utilization in ViTs for visual representation
learning. We embed our Fibottention mechanism into multiple state-of-the-art
transformer architectures dedicated to visual tasks. Leveraging only 2-6% of
the elements in the self-attention heads, Fibottention in conjunction with ViT
and its variants, consistently achieves significant performance boosts compared
to standard ViTs in nine datasets across three domains $\unicode{x2013}$ image
classification, video understanding, and robot learning tasks.

通过引入 Fibonacci 序列，将 Fibottention 机制嵌入多种视觉任务的 Transformer 架构中，可以在保持计算开销较低的同时，显著提升图像分类、视频理解和机器人学习任务中 ViT 及其变体的性能。

Fibottention: 多头多视线接纳的初级视觉表征学习

Fibottention: Inceptive Visual Representation Learning with Diverse  Attention Across Heads

Missing data in time series is a challenging issue affecting time series
analysis. Missing data occurs due to problems like data drops or sensor
malfunctioning. Imputation methods are used to fill in these values, with
quality of imputation having a significant impact on downstream tasks like
classification. In this work, we propose a semi-supervised imputation method,
ST-Impute, that uses both unlabeled data along with downstream task's labeled
data. ST-Impute is based on sparse self-attention and trains on tasks that
mimic the imputation process. Our results indicate that the proposed method
outperforms the existing supervised and unsupervised time series imputation
methods measured on the imputation quality as well as on the downstream tasks
ingesting imputed time series.

本文提出了一种半监督时间序列插值方法 ST-Impute，该方法基于稀疏自注意力模型并结合上下游任务的标签数据和无标签数据，在时间序列插值质量和下游任务结果上优于现有的监督和非监督方法。

使用半监督学习进行时间序列插值：填补丢失数据的空缺

Filling out the missing gaps: Time Series Imputation with  Semi-Supervised Learning

In this paper, we address the semantic segmentation task with a new context
aggregation scheme named \emph{object context}, which focuses on enhancing the
role of object information. Motivated by the fact that the category of each
pixel is inherited from the object it belongs to, we define the object context
for each pixel as the set of pixels that belong to the same category as the
given pixel in the image. We use a binary relation matrix to represent the
relationship between all pixels, where the value one indicates the two selected
pixels belong to the same category and zero otherwise.
We propose to use a dense relation matrix to serve as a surrogate for the
binary relation matrix. The dense relation matrix is capable to emphasize the
contribution of object information as the relation scores tend to be larger on
the object pixels than the other pixels. Considering that the dense relation
matrix estimation requires quadratic computation overhead and memory
consumption w.r.t. the input size, we propose an efficient interlaced sparse
self-attention scheme to model the dense relations between any two of all
pixels via the combination of two sparse relation matrices.
To capture richer context information, we further combine our interlaced
sparse self-attention scheme with the conventional multi-scale context schemes
including pyramid pooling~\citep{zhao2017pyramid} and atrous spatial pyramid
pooling~\citep{chen2018deeplab}. We empirically show the advantages of our
approach with competitive performances on five challenging benchmarks
including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff

本文提出一种新的上下文聚合方案 —— 物体上下文来提高物体信息的影响力，通过二元关系矩阵和稠密关系矩阵来代替对象上下文，结合多尺度上下文方案，实验结果表明该方法在语义分割任务中有着竞争性的表现。