We study the task of embodied visual active learning, where an agent is set
to explore a 3d environment with the goal to acquire visual scene understanding
by actively selecting views for which to request annotation. While accurate on
some benchmarks, today's deep visual recognition pipelines tend to not
generalize well in certain real-world scenarios, or for unusual viewpoints.
Robotic perception, in turn, requires the capability to refine the recognition
capabilities for the conditions where the mobile system operates, including
cluttered indoor environments or poor illumination. This motivates the proposed
task, where an agent is placed in a novel environment with the objective of
improving its visual recognition capability. To study embodied visual active
learning, we develop a battery of agents - both learnt and pre-specified - and
with different levels of knowledge of the environment. The agents are equipped
with a semantic segmentation network and seek to acquire informative views,
move and explore in order to propagate annotations in the neighbourhood of
those views, then refine the underlying segmentation network by online
retraining. The trainable method uses deep reinforcement learning with a reward
function that balances two competing objectives: task performance, represented
as visual recognition accuracy, which requires exploring the environment, and
the necessary amount of annotated data requested during active exploration. We
extensively evaluate the proposed models using the photorealistic Matterport3D
simulator and show that a fully learnt method outperforms comparable
pre-specified counterparts, even when requesting fewer annotations.

本文研究了具体化视觉主动学习任务，该任务通过选择性的视野请求注释来探索三维环境，以获取场景理解，我们通过开发一系列智能体来研究具体化视觉主动学习，其中智能体配备语义分割网络，并使用深度强化学习以及奖励函数平衡任务性能和必要注释数据请求，得到了可靠的结果。