A first-person camera, placed at a person's head, captures, which objects are
important to the camera wearer. Most prior methods for this task learn to
detect such important objects from the manually labeled first-person data in a
supervised fashion. However, important objects are strongly related to the
camera wearer's internal state such as his intentions and attention, and thus,
only the person wearing the camera can provide the importance labels. Such a
constraint makes the annotation process costly and limited in scalability.
In this work, we show that we can detect important objects in first-person
images without the supervision by the camera wearer or even third-person
labelers. We formulate an important detection problem as an interplay between
the 1) segmentation and 2) recognition agents. The segmentation agent first
proposes a possible important object segmentation mask for each image, and then
feeds it to the recognition agent, which learns to predict an important object
mask using visual semantics and spatial features.
We implement such an interplay between both agents via an alternating
cross-pathway supervision scheme inside our proposed Visual-Spatial Network
(VSN). Our VSN consists of spatial ("where") and visual ("what") pathways, one
of which learns common visual semantics while the other focuses on the spatial
location cues. Our unsupervised learning is accomplished via a cross-pathway
supervision, where one pathway feeds its predictions to a segmentation agent,
which proposes a candidate important object segmentation mask that is then used
by the other pathway as a supervisory signal. We show our method's success on
two different important object datasets, where our method achieves similar or
better results as the supervised methods.

本研究提出了一种基于交替网络和路径监督的无监督学习方法，不需要依赖摄像机佩戴者或第三方注释者的重要目标检测，该方法在两个不同的数据集上实现了与有监督学习类似或更好的结果。

第一人称视角视频中重要物体的无监督学习

Unsupervised Learning of Important Objects from First-Person Videos

Unlike traditional third-person cameras mounted on robots, a first-person
camera, captures a person's visual sensorimotor object interactions from up
close. In this paper, we study the tight interplay between our momentary visual
attention and motor action with objects from a first-person camera. We propose
a concept of action-objects---the objects that capture person's conscious
visual (watching a TV) or tactile (taking a cup) interactions. Action-objects
may be task-dependent but since many tasks share common person-object spatial
configurations, action-objects exhibit a characteristic 3D spatial distance and
orientation with respect to the person.
We design a predictive model that detects action-objects using EgoNet, a
joint two-stream network that holistically integrates visual appearance (RGB)
and 3D spatial layout (depth and height) cues to predict per-pixel likelihood
of action-objects. Our network also incorporates a first-person coordinate
embedding, which is designed to learn a spatial distribution of the
action-objects in the first-person data. We demonstrate EgoNet's predictive
power, by showing that it consistently outperforms previous baseline
approaches. Furthermore, EgoNet also exhibits a strong generalization ability,
i.e., it predicts semantically meaningful objects in novel first-person
datasets. Our method's ability to effectively detect action-objects could be
used to improve robots' understanding of human-object interactions.

本文利用第一视角摄像头，提出了一种称为 “action-objects” 的概念，并使用 EgoNet 设计了一个预测模型来检测 action-objects。实验表明，该模型在检测 action-objects 方面比之前的基线方法表现更好，具有较强的泛化能力。