Egocentric augmented reality devices such as wearable glasses passively
capture visual data as a human wearer tours a home environment. We envision a
scenario wherein the human communicates with an AI agent powering such a device
by asking questions (e.g., where did you last see my keys?). In order to
succeed at this task, the egocentric AI assistant must (1) construct
semantically rich and efficient scene memories that encode spatio-temporal
information about objects seen during the tour and (2) possess the ability to
understand the question and ground its answer into the semantic memory
representation. Towards that end, we introduce (1) a new task - Episodic Memory
Question Answering (EMQA) wherein an egocentric AI assistant is provided with a
video sequence (the tour) and a question as an input and is asked to localize
its answer to the question within the tour, (2) a dataset of grounded questions
designed to probe the agent's spatio-temporal understanding of the tour, and
(3) a model for the task that encodes the scene as an allocentric, top-down
semantic feature map and grounds the question into the map to localize the
answer. We show that our choice of episodic scene memory outperforms naive,
off-the-shelf solutions for the task as well as a host of very competitive
baselines and is robust to noise in depth, pose as well as camera jitter. The
project page can be found at: this https URL .

通过 Episodic Memory Question Answering (EMQA) 任务以及一系列数据集和模型的介绍，本文阐述了在面向家居环境下，使用智能语音助手等 AR 设备进行信息沟通的场景中，利用换人的机器学习算法，更好地实现 AI 视觉辅助的目标。