Accurate affordance detection and segmentation with pixel precision is an
important piece in many complex systems based on interactions, such as robots
and assitive devices. We present a new approach to affordance perception which
enables accurate multi-label segmentation. Our approach can be used to
automatically extract grounded affordances from first person videos of
interactions using a 3D map of the environment providing pixel level precision
for the affordance location. We use this method to build the largest and most
complete dataset on affordances based on the EPIC-Kitchen dataset, EPIC-Aff,
which provides interaction-grounded, multi-label, metric and spatial affordance
annotations. Then, we propose a new approach to affordance segmentation based
on multi-label detection which enables multiple affordances to co-exists in the
same space, for example if they are associated with the same object. We present
several strategies of multi-label detection using several segmentation
architectures. The experimental results highlight the importance of the
multi-label detection. Finally, we show how our metric representation can be
exploited for build a map of interaction hotspots in spatial action-centric
zones and use that representation to perform a task-oriented navigation.

准确的多标签感知能力和分割是基于交互的许多复杂系统中的重要组成部分。我们提出了一种新的感知能力方法，它能够实现准确的多标签分割。该方法可以从交互的第一人称视频中自动提取基于环境的感知能力，并提供感知能力位置的像素级精度。使用此方法构建了基于 EPIC-Kitchen 数据集的最大且最完整的感知能力数据集 EPIC-Aff，其中提供了基于交互、多标签、度量和空间感知能力注释。然后，我们提出了一种基于多标签检测的感知能力分割方法，可以使多个感知能力在同一空间中存在，例如与相同对象相关联。我们提出了几种使用多种分割架构的多标签检测策略。实验结果凸显了多标签检测的重要性。最后，我们展示了如何利用我们的度量表示来构建空间行为中心区域的交互热点地图，并使用该表示执行任务导向的导航。

多标签自我中心视觉感知映射

Multi-label affordance mapping from egocentric vision

Colorectal cancer is one of the most common cancers in the world. While
colonoscopy is an effective screening technique, navigating an endoscope
through the colon to detect polyps is challenging. A 3D map of the observed
surfaces could enhance the identification of unscreened colon tissue and serve
as a training platform. However, reconstructing the colon from video footage
remains unsolved due to numerous factors such as self-occlusion, reflective
surfaces, lack of texture, and tissue deformation that limit feature-based
methods. Learning-based approaches hold promise as robust alternatives, but
necessitate extensive datasets. By establishing a benchmark, the 2022 EndoVis
sub-challenge SimCol3D aimed to facilitate data-driven depth and pose
prediction during colonoscopy. The challenge was hosted as part of MICCAI 2022
in Singapore. Six teams from around the world and representatives from academia
and industry participated in the three sub-challenges: synthetic depth
prediction, synthetic pose prediction, and real pose prediction. This paper
describes the challenge, the submitted methods, and their results. We show that
depth prediction in virtual colonoscopy is robustly solvable, while pose
estimation remains an open research question.

结直肠癌是世界上最常见的癌症之一。本文描述了 2022 年 EndoVis 挑战赛中的人工结直肠三维重建问题及其解决方案的研究成果，并展示了虚拟结直肠镜检查中深度预测问题的可靠解决方案，同时指出姿态估计问题仍然是一个待解决的研究方向。

SimCol3D - 结肠镜检中的三维重建挑战

SimCol3D -- 3D Reconstruction during Colonoscopy Challenge

Can we relocalize in a scene represented by a single reference image?
Standard visual relocalization requires hundreds of images and scale
calibration to build a scene-specific 3D map. In contrast, we propose Map-free
Relocalization, i.e., using only one photo of a scene to enable instant, metric
scaled relocalization. Existing datasets are not suitable to benchmark map-free
relocalization, due to their focus on large scenes or their limited
variability. Thus, we have constructed a new dataset of 655 small places of
interest, such as sculptures, murals and fountains, collected worldwide. Each
place comes with a reference image to serve as a relocalization anchor, and
dozens of query images with known, metric camera poses. The dataset features
changing conditions, stark viewpoint changes, high variability across places,
and queries with low to no visual overlap with the reference image. We identify
two viable families of existing methods to provide baseline results: relative
pose regression, and feature matching combined with single-image depth
prediction. While these methods show reasonable performance on some favorable
scenes in our dataset, map-free relocalization proves to be a challenge that
requires new, innovative solutions.

提出了一种无需建立场景特定三维地图的场景单图参考图像重定位技术，并通过对世界范围内的小型艺术品，如雕塑、壁画和喷泉的构建新数据集，测试了现有算法的可行性及基准结果，并发现无地图重定位仍需新型创新技术的挑战。