Humans excel at acquiring knowledge through observation. For example, we can
learn to use new tools by watching demonstrations. This skill is fundamental
for intelligent systems to interact with the world. A key step to acquire this
skill is to identify what part of the object affords each action, which is
called affordance grounding. In this paper, we address this problem and propose
a framework called LOCATE that can identify matching object parts across
images, to transfer knowledge from images where an object is being used
(exocentric images used for learning), to images where the object is inactive
(egocentric ones used to test). To this end, we first find interaction areas
and extract their feature embeddings. Then we learn to aggregate the embeddings
into compact prototypes (human, object part, and background), and select the
one representing the object part. Finally, we use the selected prototype to
guide affordance grounding. We do this in a weakly supervised manner, learning
only from image-level affordance and object labels. Extensive experiments
demonstrate that our approach outperforms state-of-the-art methods by a large
margin on both seen and unseen objects.

本文提出一种名为 LOCATE 的框架，采用图像级别的适应性和对象标签对交互区域进行特征嵌入，并将其聚合为人类、对象部分和背景的紧凑原型之一，以指导功能接地的弱监督学习。对于新物体，实验证明我们的方法在训练图像和测试图像上性能都很好。

LOCATE: 弱监督可支配定位与传输物体部件

LOCATE: Localize and Transfer Object Parts for Weakly Supervised  Affordance Grounding

Understanding a person's behavior from their 3D motion is a fundamental
problem in computer vision with many applications. An important component of
this problem is 3D Temporal Action Localization (3D-TAL), which involves
recognizing what actions a person is performing, and when. State-of-the-art
3D-TAL methods employ a two-stage approach in which the action span detection
task and the action recognition task are implemented as a cascade. This
approach, however, limits the possibility of error-correction. In contrast, we
propose LocATe, an end-to-end approach that jointly localizes and recognizes
actions in a 3D sequence. Further, unlike existing autoregressive models that
focus on modeling the local context in a sequence, LocATe's transformer model
is capable of capturing long-term correlations between actions in a sequence.
Unlike transformer-based object-detection and classification models which
consider image or patch features as input, the input in 3D-TAL is a long
sequence of highly correlated frames. To handle the high-dimensional input, we
implement an effective input representation, and overcome the diffuse attention
across long time horizons by introducing sparse attention in the model. LocATe
outperforms previous approaches on the existing PKU-MMD 3D-TAL benchmark
(mAP=93.2%). Finally, we argue that benchmark datasets are most useful where
there is clear room for performance improvement. To that end, we introduce a
new, challenging, and more realistic benchmark dataset, BABEL-TAL-20 (BT20),
where the performance of state-of-the-art methods is significantly worse. The
dataset and code for the method will be available for research purposes.

提出了 LocATe 方法，一种全新的、端到端的 3D-TAL 方法，同时定位和识别 3D 运动中的动作，并用稀疏注意力处理高维输入，能够更好地捕捉动作之间的长期关联，并优于当前最先进的 3D-TAL 方法，在现有的 PKU-MMD 3D-TAL 基准上实现了 93.2％的 mAP，同时提出了一种具有挑战性和更实际的基准数据集 BABEL-TAL-20（BT20）。