Affordance grounding aims to localize the interaction regions for the
manipulated objects in the scene image according to given instructions. A
critical challenge in affordance grounding is that the embodied agent should
understand human instructions and analyze which tools in the environment can be
used, as well as how to use these tools to accomplish the instructions. Most
recent works primarily supports simple action labels as input instructions for
localizing affordance regions, failing to capture complex human objectives.
Moreover, these approaches typically identify affordance regions of only a
single object in object-centric images, ignoring the object context and
struggling to localize affordance regions of multiple objects in complex scenes
for practical applications. To address this concern, for the first time, we
introduce a new task of affordance grounding based on natural language
instructions, extending it from previously using simple labels for complex
human instructions. For this new task, we propose a new framework, WorldAfford.
We design a novel Affordance Reasoning Chain-of-Thought Prompting to reason
about affordance knowledge from LLMs more precisely and logically.
Subsequently, we use SAM and CLIP to localize the objects related to the
affordance knowledge in the image. We identify the affordance regions of the
objects through an affordance region localization module. To benchmark this new
task and validate our framework, an affordance grounding dataset, LLMaFF, is
constructed. We conduct extensive experiments to verify that WorldAfford
performs state-of-the-art on both the previous AGD20K and the new LLMaFF
dataset. In particular, WorldAfford can localize the affordance regions of
multiple objects and provide an alternative when objects in the environment
cannot fully match the given instruction.

基于自然语言指令的机会理解是本文的主题，通过 WorldAfford 框架中的创新性的机会推理链思维提示，使用 SAM 和 CLIP 定位与机会知识相关的对象，通过机会区域本地化模块确定对象的机会区域，在 LLMaFF 数据集上验证该框架表现出领先的状态，在多个物体上定位机会区域并在环境中无法完全匹配给定指令时提供替代方案。

WorldAfford: 基于自然语言指令的可实施性落地

WorldAfford: Affordance Grounding based on Natural Language Instructions

In this paper, we address the semantic segmentation task with a new context
aggregation scheme named \emph{object context}, which focuses on enhancing the
role of object information. Motivated by the fact that the category of each
pixel is inherited from the object it belongs to, we define the object context
for each pixel as the set of pixels that belong to the same category as the
given pixel in the image. We use a binary relation matrix to represent the
relationship between all pixels, where the value one indicates the two selected
pixels belong to the same category and zero otherwise.
We propose to use a dense relation matrix to serve as a surrogate for the
binary relation matrix. The dense relation matrix is capable to emphasize the
contribution of object information as the relation scores tend to be larger on
the object pixels than the other pixels. Considering that the dense relation
matrix estimation requires quadratic computation overhead and memory
consumption w.r.t. the input size, we propose an efficient interlaced sparse
self-attention scheme to model the dense relations between any two of all
pixels via the combination of two sparse relation matrices.
To capture richer context information, we further combine our interlaced
sparse self-attention scheme with the conventional multi-scale context schemes
including pyramid pooling~\citep{zhao2017pyramid} and atrous spatial pyramid
pooling~\citep{chen2018deeplab}. We empirically show the advantages of our
approach with competitive performances on five challenging benchmarks
including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff

本文提出一种新的上下文聚合方案 —— 物体上下文来提高物体信息的影响力，通过二元关系矩阵和稠密关系矩阵来代替对象上下文，结合多尺度上下文方案，实验结果表明该方法在语义分割任务中有着竞争性的表现。

OCNet: 场景解析中的对象上下文网络

OCNet: Object Context Network for Scene Parsing

In natural images, the scales (thickness) of object skeletons may
dramatically vary among objects and object parts, making object skeleton
detection a challenging problem. We present a new convolutional neural network
(CNN) architecture by introducing a novel hierarchical feature integration
mechanism, named Hi-Fi, to address the skeleton detection problem. The proposed
CNN-based approach has a powerful multi-scale feature integration ability that
intrinsically captures high-level semantics from deeper layers as well as
low-level details from shallower layers. % By hierarchically integrating
different CNN feature levels with bidirectional guidance, our approach (1)
enables mutual refinement across features of different levels, and (2)
possesses the strong ability to capture both rich object context and
high-resolution details. Experimental results show that our method
significantly outperforms the state-of-the-art methods in terms of effectively
fusing features from very different scales, as evidenced by a considerable
performance improvement on several benchmarks.

本论文提出了一种新的卷积神经网络架构，称为 Hi-Fi，并通过引入新的分层特征集成机制来解决对象骨架检测问题，在多种基准测试上取得了显着的性能提升，这表明了该方法的有效性。