Window attention, position embeddings, and high resolution finetuning are
core concepts in the modern transformer era of computer vision. However, we
find that naively combining these near ubiquitous components can have a
detrimental effect on performance. The issue is simple: interpolating position
embeddings while using window attention is wrong. We study two state-of-the-art
methods that have these three components, namely Hiera and ViTDet, and find
that both do indeed suffer from this bug. To fix it, we introduce a simple
absolute window position embedding strategy, which solves the bug outright in
Hiera and allows us to increase both speed and performance of the model in
ViTDet. We finally combine the two to obtain HieraDet, which achieves 61.7 box
mAP on COCO, making it state-of-the-art for models that only use ImageNet-1k
pretraining. This all stems from what is essentially a 3 line bug fix, which we
name "absolute win".

在现代计算机视觉转换器时代，窗口注意力、位置嵌入和高分辨率微调是核心概念。然而，我们发现，天真地结合这些几乎无处不在的组件可能会对性能产生不利影响。我们研究了两种具有这三个组件的最先进方法，即 Hiera 和 ViTDet，发现两者都确实遭受了这个错误的困扰。为了解决这个问题，我们引入了一种简单的绝对窗口位置嵌入策略，在 Hiera 中彻底解决了该错误，并允许我们在 ViTDet 中提高模型的速度和性能。最后，我们将两者结合起来得到 HieraDet，在 COCO 上实现了 61.7 的盒子 mAP，成为仅使用 ImageNet-1k 预训练的模型的最先进技术。这一切都源自本质上是一个三行错误修复，我们将其命名为 “绝对获胜”。

窗口关注存在漏洞：如何不插入位置嵌入

Window Attention is Bugged: How not to Interpolate Position Embeddings

Deploying Large Language Models (LLMs) in streaming applications such as
multi-round dialogue, where long interactions are expected, is urgently needed
but poses two major challenges. Firstly, during the decoding stage, caching
previous tokens' Key and Value states (KV) consumes extensive memory. Secondly,
popular LLMs cannot generalize to longer texts than the training sequence
length. Window attention, where only the most recent KVs are cached, is a
natural approach -- but we show that it fails when the text length surpasses
the cache size. We observe an interesting phenomenon, namely attention sink,
that keeping the KV of initial tokens will largely recover the performance of
window attention. In this paper, we first demonstrate that the emergence of
attention sink is due to the strong attention scores towards initial tokens as
a ``sink'' even if they are not semantically important. Based on the above
analysis, we introduce StreamingLLM, an efficient framework that enables LLMs
trained with a finite length attention window to generalize to infinite
sequence lengths without any fine-tuning. We show that StreamingLLM can enable
Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language
modeling with up to 4 million tokens and more. In addition, we discover that
adding a placeholder token as a dedicated attention sink during pre-training
can further improve streaming deployment. In streaming settings, StreamingLLM
outperforms the sliding window recomputation baseline by up to 22.2x speedup.
Code and datasets are provided at this https URL

部署大型语言模型（LLMs）在流式应用中的一个研究论文，介绍了两个主要挑战和一个有效的解决方案 StreamingLLM，用于长文本的流式部署，能够在无需微调的情况下实现 LLMs 的稳定和高效的语言建模。

注意力汇聚的高效流式语言模型

Efficient Streaming Language Models with Attention Sinks

We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone
network for object detection. This design enables the original ViT architecture
to be fine-tuned for object detection without needing to redesign a
hierarchical backbone for pre-training. With minimal adaptations for
fine-tuning, our plain-backbone detector can achieve competitive results.
Surprisingly, we observe: (i) it is sufficient to build a simple feature
pyramid from a single-scale feature map (without the common FPN design) and
(ii) it is sufficient to use window attention (without shifting) aided with
very few cross-window propagation blocks. With plain ViT backbones pre-trained
as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the
previous leading methods that were all based on hierarchical backbones,
reaching up to 61.3 AP_box on the COCO dataset using only ImageNet-1K
pre-training. We hope our study will draw attention to research on
plain-backbone detectors. Code for ViTDet is available in Detectron2.

本研究探讨使用层次结构大减的视觉变换器（ViT）作为物体检测的骨干网络，使用最小调整实现简单的特征金字塔和窗口注意力，预训练和微调使用 ViT 和掩码自编码器，提出的模型在 COCO 数据集上达到 61.3 AP_box，与使用层次骨干网络的现有领先方法相媲美。