Unlike traditional third-person cameras mounted on robots, a first-person
camera, captures a person's visual sensorimotor object interactions from up
close. In this paper, we study the tight interplay between our momentary visual
attention and motor action with objects from a first-person camera. We propose
a concept of action-objects---the objects that capture person's conscious
visual (watching a TV) or tactile (taking a cup) interactions. Action-objects
may be task-dependent but since many tasks share common person-object spatial
configurations, action-objects exhibit a characteristic 3D spatial distance and
orientation with respect to the person.
We design a predictive model that detects action-objects using EgoNet, a
joint two-stream network that holistically integrates visual appearance (RGB)
and 3D spatial layout (depth and height) cues to predict per-pixel likelihood
of action-objects. Our network also incorporates a first-person coordinate
embedding, which is designed to learn a spatial distribution of the
action-objects in the first-person data. We demonstrate EgoNet's predictive
power, by showing that it consistently outperforms previous baseline
approaches. Furthermore, EgoNet also exhibits a strong generalization ability,
i.e., it predicts semantically meaningful objects in novel first-person
datasets. Our method's ability to effectively detect action-objects could be
used to improve robots' understanding of human-object interactions.

本文利用第一视角摄像头，提出了一种称为 “action-objects” 的概念，并使用 EgoNet 设计了一个预测模型来检测 action-objects。实验表明，该模型在检测 action-objects 方面比之前的基线方法表现更好，具有较强的泛化能力。