The emergence of attention-based transformer models has led to their
extensive use in various tasks, due to their superior generalization and
transfer properties. Recent research has demonstrated that such models, when
prompted appropriately, are excellent for few-shot inference. However, such
techniques are under-explored for dense prediction tasks like semantic
segmentation. In this work, we examine the effectiveness of prompting a
transformer-decoder with learned visual prompts for the generalized few-shot
segmentation (GFSS) task. Our goal is to achieve strong performance not only on
novel categories with limited examples, but also to retain performance on base
categories. We propose an approach to learn visual prompts with limited
examples. These learned visual prompts are used to prompt a multiscale
transformer decoder to facilitate accurate dense predictions. Additionally, we
introduce a unidirectional causal attention mechanism between the novel
prompts, learned with limited examples, and the base prompts, learned with
abundant data. This mechanism enriches the novel prompts without deteriorating
the base class performance. Overall, this form of prompting helps us achieve
state-of-the-art performance for GFSS on two different benchmark datasets:
COCO-$20^i$ and Pascal-$5^i$, without the need for test-time optimization (or
transduction). Furthermore, test-time optimization leveraging unlabelled test
data can be used to improve the prompts, which we refer to as transductive
prompt tuning.

通过使用学习到的视觉提示，我们的工作研究了在少样本情况下，通过对 Transformer 解码器进行提示，来提高普适少样本分割（GFSS）任务的效果。我们提出了一种利用少量样本学习视觉提示的方法，通过这些学习到的视觉提示，我们可以对多尺度 Transformer 解码器进行提示，以便实现准确的密集预测。此外，我们引入了一种单向因果关注机制，用于连接通过少样本学习到的新提示和通过大量数据学习到的基础提示，从而提升新提示的质量而不损害基础类别的性能。总体而言，这种提示形式帮助我们在两个不同的基准数据集上实现了 GFSS 的最先进性能：COCO-$20^i$ 和 Pascal-$5^i$，而无需进行测试时间的优化或传导。此外，我们还使用未标记的测试数据进行测试时间的优化，以改进这些视觉提示，我们称之为传导提示调优（transductive prompt tuning）。

广义少样本分割的视觉引导：多尺度方法

Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale  Approach

While attention is all you need may be proving true, we do not know why:
attention-based transformer models such as BERT are superior but how
information flows from input tokens to output predictions are unclear. We
introduce influence patterns, abstractions of sets of paths through a
transformer model. Patterns quantify and localize the flow of information to
paths passing through a sequence of model nodes. Experimentally, we find that
significant portion of information flow in BERT goes through skip connections
instead of attention heads. We further show that consistency of patterns across
instances is an indicator of BERT's performance. Finally, We demonstrate that
patterns account for far more model performance than previous attention-based
and layer-based methods.

介绍了影响模式来理解注意力机制，发现 BERT 模型中信息流主要通过跳跃连接而非注意力头，而且模式的一致性是 BERT 性能的一个指标，比以前的注意力和层次方法更好。