This paper introduces a Transformer-based integrative feature and cost
aggregation network designed for dense matching tasks. In the context of dense
matching, many works benefit from one of two forms of aggregation: feature
aggregation, which pertains to the alignment of similar features, or cost
aggregation, a procedure aimed at instilling coherence in the flow estimates
across neighboring pixels. In this work, we first show that feature aggregation
and cost aggregation exhibit distinct characteristics and reveal the potential
for substantial benefits stemming from the judicious use of both aggregation
processes. We then introduce a simple yet effective architecture that harnesses
self- and cross-attention mechanisms to show that our approach unifies feature
aggregation and cost aggregation and effectively harnesses the strengths of
both techniques. Within the proposed attention layers, the features and cost
volume both complement each other, and the attention layers are interleaved
through a coarse-to-fine design to further promote accurate correspondence
estimation. Finally at inference, our network produces multi-scale predictions,
computes their confidence scores, and selects the most confident flow for final
prediction. Our framework is evaluated on standard benchmarks for semantic
matching, and also applied to geometric matching, where we show that our
approach achieves significant improvements compared to existing methods.

本论文介绍了一种基于 Transformer 的综合特征和成本聚合网络，用于密集匹配任务。我们展示了特征聚合和成本聚合的不同特点，并介绍了一种简单而有效的架构，利用自注意力机制和互注意力机制来统一特征聚合和成本聚合，从而在语义匹配和几何匹配任务中实现了显著的改进。

用 Transformer 统一特征和成本聚合的语义和视觉对应

Unifying Feature and Cost Aggregation with Transformers for Semantic and  Visual Correspondence

This paper presents a novel cost aggregation network, called Volumetric
Aggregation with Transformers (VAT), for few-shot segmentation. The use of
transformers can benefit correlation map aggregation through self-attention
over a global receptive field. However, the tokenization of a correlation map
for transformer processing can be detrimental, because the discontinuity at
token boundaries reduces the local context available near the token edges and
decreases inductive bias. To address this problem, we propose a 4D
Convolutional Swin Transformer, where a high-dimensional Swin Transformer is
preceded by a series of small-kernel convolutions that impart local context to
all pixels and introduce convolutional inductive bias. We additionally boost
aggregation performance by applying transformers within a pyramidal structure,
where aggregation at a coarser level guides aggregation at a finer level. Noise
in the transformer output is then filtered in the subsequent decoder with the
help of the query's appearance embedding. With this model, a new
state-of-the-art is set for all the standard benchmarks in few-shot
segmentation. It is shown that VAT attains state-of-the-art performance for
semantic correspondence as well, where cost aggregation also plays a central
role.

本文提出了一种新的成本聚合网络，称为带有 Transformers（VAT）的体积聚合，用于少量样本分割，并通过 4D 卷积 Swin Transformer 解决了 Transformer 处理相关矩阵时可能遇到的问题，实现了在所有标准基准测试中的最新技术成果，且在语义对应性方面的表现也非常优异。

基于 4D 卷积 Swin Transformer 的成本汇聚用于少样本分割

Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot  Segmentation

We introduce a novel cost aggregation network, dubbed Volumetric Aggregation
with Transformers (VAT), to tackle the few-shot segmentation task by using both
convolutions and transformers to efficiently handle high dimensional
correlation maps between query and support. In specific, we propose our encoder
consisting of volume embedding module to not only transform the correlation
maps into more tractable size but also inject some convolutional inductive bias
and volumetric transformer module for the cost aggregation. Our encoder has a
pyramidal structure to let the coarser level aggregation to guide the finer
level and enforce to learn complementary matching scores. We then feed the
output into our affinity-aware decoder along with the projected feature maps
for guiding the segmentation process. Combining these components, we conduct
experiments to demonstrate the effectiveness of the proposed method, and our
method sets a new state-of-the-art for all the standard benchmarks in few-shot
segmentation task. Furthermore, we find that the proposed method attains
state-of-the-art performance even for the standard benchmarks in semantic
correspondence task although not specifically designed for this task. We also
provide an extensive ablation study to validate our architectural choices. The
trained weights and codes are available at: this https URL

介绍了一种新颖的成本聚合网络 ——Volumetric Aggregation with Transformers (VAT)，通过使用卷积和 Transformer 处理查询和支持之间的高维相关性地图，从而有效地处理 Few-shot 分割任务。