Egocentric sensors such as AR/VR devices capture human-object interactions
and offer the potential to provide task-assistance by recalling 3D locations of
objects of interest in the surrounding environment. This capability requires
instance tracking in real-world 3D scenes from egocentric videos (IT3DEgo). We
explore this problem by first introducing a new benchmark dataset, consisting
of RGB and depth videos, per-frame camera pose, and instance-level annotations
in both 2D camera and 3D world coordinates. We present an evaluation protocol
which evaluates tracking performance in 3D coordinates with two settings for
enrolling instances to track: (1) single-view online enrollment where an
instance is specified on-the-fly based on the human wearer's interactions. and
(2) multi-view pre-enrollment where images of an instance to be tracked are
stored in memory ahead of time. To address IT3DEgo, we first re-purpose methods
from relevant areas, e.g., single object tracking (SOT) -- running SOT methods
to track instances in 2D frames and lifting them to 3D using camera pose and
depth. We also present a simple method that leverages pretrained segmentation
and detection models to generate proposals from RGB frames and match proposals
with enrolled instance images. Perhaps surprisingly, our extensive experiments
show that our method (with no finetuning) significantly outperforms SOT-based
approaches. We conclude by arguing that the problem of egocentric instance
tracking is made easier by leveraging camera pose and using a 3D allocentric
(world) coordinate representation.

借助自我中心传感器，本研究提出了一种新方法来解决三维视角下的物体实例追踪问题，并证明相比二维追踪方法，基于相机姿态和三维坐标的方法显著提高了追踪性能。

从自我中心视频中的三维场景中实例追踪

Instance Tracking in 3D Scenes from Egocentric Videos

Tracking segmentation masks of multiple instances has been intensively
studied, but still faces two fundamental challenges: 1) the requirement of
large-scale, frame-wise annotation, and 2) the complexity of two-stage
approaches. To resolve these challenges, we introduce a novel semi-supervised
framework by learning instance tracking networks with only a labeled image
dataset and unlabeled video sequences. With an instance contrastive objective,
we learn an embedding to discriminate each instance from the others. We show
that even when only trained with images, the learned feature representation is
robust to instance appearance variations, and is thus able to track objects
steadily across frames. We further enhance the tracking capability of the
embedding by learning correspondence from unlabeled videos in a self-supervised
manner. In addition, we integrate this module into single-stage instance
segmentation and pose estimation frameworks, which significantly reduce the
computational complexity of tracking compared to two-stage networks. We conduct
experiments on the YouTube-VIS and PoseTrack datasets. Without any video
annotation efforts, our proposed method can achieve comparable or even better
performance than most fully-supervised methods.

本文介绍了一种利用标记图像数据和未标记视频序列学习实例跟踪网络的半监督框架，采用实例对比目标，利用学习的嵌入区分每个实例并在不同帧之间稳定地跟踪对象，并且将此模块集成到单阶段实例分割和姿态估计框架中，较之两阶段网络显著降低了跟踪的计算复杂度。无需任何视频标注努力，我们的方法可达到与大多数完全监督方法相当甚至更好的性能。